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Abstract 

In  December  1996,  the  Cost/Schedule  Control  Systems  Criteria  (C/SCSC)  was 
officially  replaced  by  the  Earned  Value  Management  Systems  (EVMS)  criteria.  The 
switch  to  EVMS,  coupled  with  current  acquisition  reform  changes,  have  left  many 
wondering  what  the  effects  of  these  changes  will  be. 

This  thesis  defines  the  costs  and  benefits  of  the  old  C/SCSC,  and  then  compares 
them.  Additionally,  this  thesis  discusses  the  changes  accompanying  the  switch  to  EVMS 
and  the  effect  on  the  costs  and  benefits. 

The  marginal  costs  of  C/SCSC  are  defined  as  the  difference  between  the  costs  of  a 
C/SCSC-compliant  system  and  a  contractors  ‘normal’  management  control  system.  The 
marginal  system  compliance  costs  are  334  -  481  person  days,  while  the  marginal 
operating  costs  are  50%  of  the  C/SCSC-compliant  operating  costs. 

Fourteen  benefits  of  C/SCSC  are  detailed  in  this  thesis.  The  most  important 
benefit  discovered  was  the  data  reliability  that  comes  with  a  criteria-compliant 
management  control  system. 

The  main  difference  between  C/SCSC  and  EVMS  is  the  system  certification 
process.  Under  C/SCSC,  DoD  teams  would  have  to  certify  a  contraetor’s  system.  Under 
EVMS,  contractors  have  the  ability  to  self-certify  their  system  (with  final  government 
approval).  Cost  savings  may  result  through  self-certification  without  reductions  in  the 
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benefits. 


A  COST-BENEFIT  ANALYSIS  OF  EARNED  VALUE 


MANAGEMENT  SYSTEM  CRITERIA 


I.  Introduction 


General  Issue 

Since  1967,  the  Department  of  Defense  (DOD)  has  required  contractors  to  comply 
with  Cost/Schedule  Control  Systems  Criteria  (C/SCSC)  on  “significanf’  DOD  contracts. 
These  criteria,  now  termed  Earned  Value  Management  System  Criteria,  are  nothing  more 
than  standards  intended  to  ensure  that  contractors  use  sound  business  practices.  Reports 
from  criteria-compliant  contractors  provide  insight  into  the  cost,  schedule,  and  technical 
progress  of  the  contract. 

Although  there  have  been  several  reports  sponsored  by  DOD  and  private  industry 
regarding  either  the  costs  or  the  benefits  of  these  standards  (Arthur  D.  Little  Study  and 
Coopers  &  Lybrand/TASC  Study  are  two),  we  could  find  no  study  which  objectively 
compared  the  cost  with  the  benefits.  It  is  not  difficult  to  understand  how  these  reports 
could  be  biased.  In  fact,  it  is  easy  to  understand  why  a  report  sponsored  by  the  private 
sector  regarding  the  usefulness  of  the  criteria  may  be  biased  by  a  desire  for  less 
government  oversight.  Likewise,  a  report  sponsored  by  the  government  may  be 
influenced  by  the  government’s  legitimate  need  for  that  oversight. 


We  are  not  insinuating  that  any  particular  report  is  purposely  incomplete  or 
biased.  However,  we  do  suggest  the  undertaking  of  a  more  thorough  and  fair  analysis 
that  examines  both  the  costs  and  the  benefits  of  the  criteria  at  the  same  time.  Otherwise, 
a  reader  may  be  inappropriately  persuaded  that  the  costs  exceed  the  benefits  of  the  criteria 
or  vise-versa. 

This  thesis  perhaps  provides  the  first  objective  comparison  of  the  costs  and 
benefits  of  the  criteria.  Due  to  shrinking  defense  budgets,  acquisition  reform,  and  the 
need  to  be  as  efficient  as  possible,  we  believe  this  comparison  is  critical  for  the  DOD 
acquisition  community.  Many  DOD  sponsored  studies  have  looked  at  either  the  costs  or 
the  benefits,  but  none  has  taken  an  objective  look  at  both.  In  order  for  decision  makers  to 
be  better  informed  when  participating  in  future  earned  value  streamlining  decisions,  an 
objective  comparison  of  costs  and  benefits  will  be  extremely  beneficial. 


Background 

It  is  important  to  note  that  industry’s  32  Earned  Value  Management  System 
(EVMS)  criteria  replaced  the  DOD’s  35  Cost/Schedule  Control  Systems  Criteria 
(C/SCSC)  in  December  1996.  In  this  thesis  we  use  the  term  “criteria”  to  refer  to  either 
EVMS  or  C/SCSC  criteria.  We  make  a  distinction  between  the  two  only  when  it  is 
necessary.  We  will  describe  the  origin  and  differences  between  the  former  and  revised 
criteria  later.  With  few  exceptions,  all  “significant”  DOD  contracts  must  meet  the 
criteria. 
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Unless  waived  by  the  Milestone  Decision  Authority  or  a  designated 
representative,  compliance  with  the  criteria  shall  be  required  on  significant 
contracts  and  subcontracts  within  all  acquisition  programs,  including  highly 
sensitive  classified  programs  and  major  construction  programs.  This  also  includes 
significant  contracts  executed  for  foreign  governments  and  for  specialized 
organizations  such  as  the  Defense  Advanced  Research  Projects  Agency,  and 
significant  acquisition  effort  performed  by  Government  activities.  Significant 
contracts  include  research,  development,  test,  and  evaluation  contracts  and 
subcontracts  with  a  value  of  $70  million  or  more,  or  procurement  contracts  and 
subcontracts  with  a  value  of  $300  million  or  more  (in  FY  1996  constant  dollars). 
Compliance  with  the  criteria  on  contracts  and  subcontracts  below  these  thresholds 
may  be  required  when,  in  the  DOD  component  manager's  judgment,  the  contract 
risk  or  management  interest  requires  assurance  that  the  contractor's  cost  and 
schedule  management  control  systems  are  acceptable.  (DOD  5000.2-R,  3. 3. 4.3) 


For  contracts  which  must  comply  with  the  criteria,  the  Cost  Performance  Report 
(CPR)  is  also  usually  required.  The  monthly  CPR  summarizes  the  cost,  schedule,  and 
technical  status  of  the  contract,  and  is  a  key  document  used  by  government  financial, 
contractual,  and  program  managers.  On  contracts  that  are  not  significant,  the 
Cost/Schedule  Status  Report  (C/SSR)  may  be  used.  The  C/SSR  also  summarizes  the 
cost,  schedule,  and  technical  performance  on  the  contract,  but  is  less  detailed  and  less 
reliable  than  the  CPR. 

Compliance  with  the  criteria  is  usually  not  required  on  firm  fixed  price  contracts 
(including  firm  fixed  price  contracts  with  economic  price  adjustment  provisions),  time 
and  materials  contracts,  and  contracts  which  consist  mostly  of  level-of-effort  work. 
Exceptions  may  be  made  by  the  Milestone  Decision  Authority  (MDA)  for  individual 
contracts  (DOD  5000.2-R,  3.3.4. 3). 

The  criteria  concept  does  not  mandate  a  specific  management  control  system  for  a 
contractor.  Instead,  the  concept  requires  a  management  control  system  that  complies  with 
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the  criteria.  A  contractor  is  free  to  use  any  management  control  system  which  meets  the 
criteria.  Since  1967,  most  major  defense  contractors  have  developed  criteria-compliant 
management  control  systems.  Although  a  full  explanation  is  beyond  the  scope  of  this 
thesis,  the  criteria  can  be  summarized  into  five  major  areas:  1)  organization,  2)  planning 
and  budgeting,  3)  accounting,  4)  analysis,  and  5)  revisions  and  access  to  data. 

The  first  area  deals  with  organization  and  has  five  criteria  which  define  the 
cost/schedule  system  requirements.  Here,  the  contractor  is  required  to  use  an  integrated 
management  control  system  to  define  the  contractual  effort  with  the  use  of  a  contract 
work  breakdown  structure  (CWBS),  and  assign  responsibilities  for  work  performance 
(Fleming,  1992;  33-40).  A  key  criterion  under  this  area  is  the  contractor’s  requirement  to 
“assign  responsibility  for  identified  work  to  the  appropriate  functional  area”  (Gadeken, 
1983:  16). 

The  second  area  of  the  criteria  deals  with  planning  and  budgeting.  Ten  criteria 
(eleven  under  C/SCSC)  spell  out  the  requirements  to  establish  and  maintain  a  “budget” 
baseline  for  control  of  the  work  (Fleming,  1992:  41-51).  All  work  defined  in  the  CWBS 
must  be  scheduled,  budgeted,  and  authorized.  This  leads  to  the  “establishment  of  a  time- 
phased  budget  baseline  known  as  the  performance  measurement  baseline”  (Gadeken, 
1983:  16). 

The  third  area  of  the  criteria  deals  with  accounting  requirements.  In  order  to 
satisfy  the  six  criteria  (seven  under  C/SCSC)  in  this  area,  the  contractor  must 
satisfactorily  demonstrate  the  ability  to  accumulate  costs  of  work  and  materials  in  a 
manner  that  allows  for  comparison  with  earned  value.  Earned  value  is  probably  the  most 
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important  aspect  of  the  criteria  approach  to  program  management.  Instead  of  using  the 
old  budget  versus  actual  method,  earned  value  requires  the  quantification  of  a  program’s 
progress,  using  objective  indicators  of  work  performed.  Looking  at  the  work  actually 
accomplished  during  a  given  period,  leads  way  to  realistic  assessments  of  cost  and 
schedule  performance  (Fleming,  1992:  53-58). 

The  fourth  area  of  the  criteria  lists  six  criteria  dealing  with  analysis.  These 
criteria  require  contractors  to  measure  the  earned  value,  analyze  cost  and  schedule 
variances,  and  develop  reliable  estimates  of  costs  at  completion  (Fleming,  1992:  61-67). 
Specifically,  these  criteria  detail  the  data  that  must  be  derived  from  the  contractors 
cost/schedule  control  system  (Gadeken,  1983:  16). 

The  last  area  of  the  criteria  deals  with  revisions  and  data  access.  Five  criteria  (six 
under  C/SCSC)  are  provided  that  spell  out  how  to  incorporate  changes  to  the  controlled 
baseline,  when  required,  and  allow  appropriate  government  representatives  to  have  access 
to  the  contractual  data  for  determining  criteria  compliance  (Fleming,  1992:  69-74). 

The  criteria  should  not  be  confused  with  earned  value  management.  One  does  not 
require  the  other.  Although  the  criteria  can  improve  the  reliability  of  earned  value 
management  data,  the  two  are  quite  distinct.  Indeed,  Fleming  and  Koppleman  advocate 
the  use  of  earned  value  on  almost  any  contract  or  project,  and  suggest  that  the  criteria  are 
necessary  only  on  large,  cost-reimbursable  government  contracts. 

Earned  value  management  came  about  because  governmental  buyers,  before  the 
days  when  C/SCSC  was  required,  had  no  viable  means  with  which  to  verify:  1)  how 
much  money  and  time  a  particular  job  would  likely  take,  before  starting  it;  2)  once  the 
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work  was  started,  what  physical  work  was  being  accomplished  against  the  money  being 
spent;  and  3)  what  the  total  cost  of  the  project  would  most  likely  be,  as  well  as  how  long 
completing  the  project  would  take  (Fleming,  1992:  22). 

In  order  to  try  and  link  planned  events  and  tasks  and  show  relationships  and 
constraints  between  them,  the  defense  community  came  up  with  PERT/CPM. 

PERT/CPM  was  used  very  little  at  first.  Then  the  Secretary  of  Defense  mandated  that 
this  single  management  technique  be  used  on  all  major  defense  contracts  (Fleming,  1992: 
23). 

This  approach  didn’t  work  for  several  reasons,  the  most  obvious  being  that 
managers  despised  being  told  exactly  how  to  do  their  business.  Then,  in  1967,  DOD 
officially  implemented  C/SCSC  which  defined  35  criteria  or  standards  imposed  on  all 
significant  defense  contracts.  This  new  method  didn’t  require  a  specific  system.  It 
simply  specified  the  minimum  requirements  which  a  contractor’s  management  control 
system  must  satisfy.  The  two  primary  objectives  were:  1 )  for  contractors  to  use  effective 
internal  cost  and  schedule  management  control  systems,  and  2)  for  the  government  to 
have  timely,  auditable  data  for  determining  contract  status  (Fleming,  1992:  25). 

C/SCSC  was  used  for  nearly  thirty  years,  but  was  it  effective?  Arthur  D.  Little 
Incorporated  performed  a  thorough  two  year  study  on  the  utility  of  C/SCSC.  The  study 
found  that  “C/SCSC-accepted  systems  are  considered  to  be  effective  in  assisting  to 
control  cost  and  schedule  performance.”  (1983:  IV-2).  Additionally,  this  study  found  that, 
‘‘C/SCSC  benefits  are  considered  to  outweigh  the  costs.  Nevertheless,  most  contractor 
program  managers  see  a  need  for  minor  modifications  to  their  systems.  Many 
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government  program  managers  agree”  (1983 :  IV-2).  It  is  important  to  note  that  this 
study  did  not  objectively  measure  costs  and  benefits,  but  instead  subjectively  addressed 
the  costs  and  benefits  on  the  basis  of  respondents’  perceptions.  This  study  validated  the 
need  for  C/SCSC,  but  was  C/SCSC  the  most  cost  effective  way  to  implement  earned 
value? 

Initiated  in  a  response  to  a  request  by  the  Deputy  Secretary  of  Defense,  the 
Coopers  &  Lybrand/TASC  study  undertook  the  task  of  reviewing  the  impact  of  the 
DOD’s  acquisition  regulations  and  oversight  requirements  on  defense  contractors.  The 
study  identified  over  120  regulatory  and  statutory  cost  drivers  that  total  18%  of  value 
added  costs  (value  added  is  equal  to  total  costs  less  material  costs).  The  study 
concentrated  on  the  top  ten  cost  drivers,  of  which,  C/SCSC  ranked  third.  The  study 
concluded  that,  “In  general,  industry  views  the  general  framework  and  principles  of 
cost/schedule  reporting  positively”  (1994:  22).  Industry  also  commented  that,  “  . . . 
cost/schedule  reporting  is  too  detailed,  repetitive,  and  voluminous  to  be  used  effectively 
as  a  management  tool  by  either  the  government  or  industry,  and  that  the  requirement  may 
in  fact  undermine  program  performance  by  diverting  the  time  and  attention  of  the 
company  program  manager”  (1994:  22).  Although  this  study  identified  total  C/SCSC 
costs,  it  did  not  address  the  C/SCSC  benefits.  Additionally,  the  costs  of  a  contractor’s 
normal  cost/schedule  control  system  were  not  compared  to  the  incremental  cost  of  the 
C/SCSC-compliant  system. 

A  December  1 994  memorandum  from  the  director  of  acquisition  program 
integration  tasked  all  Service  Acquisition  Executives  to  begin  work  on  a  commercial 
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cost/schedule  performance  standard.  In  addition  to  movements  within  the  DOD,  industry 
also  began  changing.  In  1995,  Lockheed  Martin  created  a  performance  management  task 
force  to  ’^expand  the  use  of  earned  value  in  the  program  management  process”  (Papers 
and  Presentations,  1997:  1).  The  task  force  is  to  review  Lockheed’s  earned  value 
management  process  (C/SCSC  validated)  and  identify  non-value  added  portions.  This 
pioneering  effort  from  industry  was  encouraged  by  the  Under  Secretary  of  Defense’s 
(Acquisition  &  Technology)  [USD(A&T)],  Dr.  Kaminski  (Papers  and  Presentations, 

1997:  1). 

Another  evolutionary  step  associated  with  the  change  to  EVMS  was  Dr. 
Kaminski's  December  1995  memorandum  which  abolished  the  Performance 
Measurement  Joint  Executive  Group  (PMJEG).  The  PMJEG  had  been  chartered  to  ensure 
consistent  implementation  of  the  C/SCSC  throughout  the  three  military  services  (Papers 
and  Presentations,  1997:  1).  The  responsibilities  of  the  PMJEG  now  reside  with  the 
Defense  Contract  Management  Command  (DCMC),  as  the  executive  agent  for  C/SCSC 
(Abba.  1997:  62). 

In  August  1996,  industry  published  the  Industry  Standard  Guidelines  for  Earned 
Value  Management  Systems  (GAO,  1997:  16).  This  guide  is  for  the  “establishment  and 
application  of  an  integrated  management  system  that  coordinates  work  scope,  schedule, 
and  cost  objectives,  and  employ’s  earned  value  methods  for  program  planning  and 
control”  {Industry  Standard  Guidelines  for  Earned  Value  Management  Systems,  1996: 

1 ).  The  guide  discusses  the  benefits  of  an  earned  value  system  and  then  details  the 
guidelines. 
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The  major  difference  between  the  C/SCSC  and  the  EVMS  is  the  system 
certification.  Under  the  C/SCSC,  a  DOD  team  would  visit  the  defense  contractor  seeking 
certification,  and  perform  a  review  to  check  for  compliance  with  the  C/SCSC.  The 
EVMS  certification  process  can  now  be  accomplished  in  two  ways:  either  self- 
certification  or  extension  of  current  C/SCSC  certification.  The  self  certification  involves 
written  corporate  assurance  that  the  defense  contractor’s  program  management  system 
meets  the  full  intentions  of  the  guidelines.  The  defense  contractor  is  responsible  for  the 
evaluation,  and  the  certificate  of  compliance  should  be  signed  by  the  Chief  Executive 
Officer  of  the  company.  Industry  may  seek  the  help  of  DOD  organizations  to  perform  or 
assist  in  these  certifications,  but  industry  controls  DOD  access  to  this  process.  {Industry 
Standard  Guidelines  for  Earned  Value  Management  Systems,  1996:  Section  5). 


Specific  Objective 

Due  to  downsizing,  budget  cuts,  and  acquisition  reform  there  have  been  many 
changes  in  the  acquisition  process.  The  benefits  of  the  criteria  have  been  tested  over  time 
and  much  has  been  written,  separately,  about  the  costs  or  benefits  of  the  criteria.  Because 
we  believe  that  little  can  be  learned  by  examining  either  side  in  isolation,  we  will 
summarize  the  costs  and  benefits  together.  Further,  we  attempt  to  predict  how  the  costs 
and  benefits  will  be  affected  by  the  new  industry  guidelines  for  EVMS. 

This  research  will  provide  a  summary  of  the  costs  and  benefits  of  C/SCSC  as 
documented  since  1967.  Additionally,  this  research  will  provide  a  compilation  of 
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important  documents,  policy  memorandums,  and  other  sources  related  to  implementing 
EVMS.  Research  in  this  area  has  not  yet  been  performed  and  will  provide  a  single  source 
of  earned  value  information. 


Research  Questions 

1)  What  are  the  costs  of  the  criteria?  Although  it  is  difficult  to  separate  the  cost  of  using 
the  criteria  from  the  cost  of  “good  business  practices,”  we  will  try  to  examine  these 
costs.  Clearly,  without  a  sense  of  the  costs  required  to  use  the  standards  we  cannot 
assess  the  overall  utility  of  the  requirement. 

2)  WTrat  are  the  benefits  of  the  criteria?  An  objective  assessment  of  the  benefits  to  both 
the  government  and  contractor  is  also  essential  to  determine  the  overall  value  of  these 
standards. 

3)  How  are  the  costs  and  benefits  related?  Based  on  our  research  we  will  attempt  to 
measure  the  overall  utility  of  the  standards.  The  answer  to  this  question  is  crucial  and 
will  become  more  important  in  the  future.  Since  the  military,  and  specifically  the 
acquisition  budgets  are  continually  declining,  many  processes  and  requirements  are 
being  scrutinized  to  determine  their  necessity.  No  doubt,  any  non-value  added 
activity  will  eventually  be  eliminated.  If  our  findings  justify  the  value  of  the 
standards,  then  this  result  may  serve  to  help  ward  off  any  attempt  to  eliminate  this 
requirement.  If  our  research  indicates  that  these  standards  are  not  cost  effective,  then 
perhaps  we  should  reconsider  requiring  these  standards. 
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4)  How  will  the  move  to  EVMS  affect  these  costs  and  benefits?  We  will  take  a  look  at 
the  new  standards  to  try  to  understand  how/if  they  will  have  an  impact  on  the  costs 
and  benefits  described  earlier. 

Description  of  the  Remaining  Research 

The  next  chapter  of  this  thesis  will  examine  all  known  studies  on  the  costs  of  the 
criteria.  Reviews  of  these  studies  will  hopefully  provide  a  consolidated  einswer  to  the 
first  research  question.  The  third  chapter  will  examine  all  known  studies  on  the  benefits 
of  the  criteria.  Reviews  of  these  studies  will  hopefully  provide  a  consolidated  answer  to 
the  second  research  question.  The  fourth  chapter  has  two  purposes.  First,  the  costs  and 
benefits  of  the  criteria  will  be  compared  in  an  effort  to  answer  the  third  research  question, 
The  second  part  of  the  fourth  chapter  will  be  a  compilation  of  information  documenting 
the  switch  from  C/SCSC  to  EVMS  criteria.  This  analysis  will  answer  the  last  research 
question.  Finally,  we  will  summarize  all  of  the  research  and  findings  in  the  fifth  chapter. 
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II.  The  Cost  ofC/SCSC 


0\’erview 


What  is  the  cost  associated  with  implementing  C/SCSC?  This  is  a  question  that 

can  not  easily  be  answered.  One  must  understand  what  is  being  asked  before  the  answer 

can  be  given.  A  better  question  might  be,  “  . . .  what  are  the  additional  costs  incurred  as  a 

consequence  of  using  the  concepts  and  principles  embodied  in  the  Criteria?”  (Brodkorb, 

1992:1).  Another  similar  question  asks  what  are  the  “  .  . .  costs  uniquely  attributable  to 

C/SCSC  such  as  support  of  validation  reviews  and  system  description  documentation?” 

(Gadeken,  1983;  1 5).  For  the  purpose  of  this  thesis,  we  want  to  separate  the  normal 

operating  costs  of  a  contractor's  management  control  system  from  the  total  cost  of 

operating  a  C/SC  SC-compliant  system  (Lampkin,  1992:  3). 

Before  doing  this,  we  first  must  address  what  drives  C/SCSC  compliance  and 

reporting  costs.  The  DOD’s  5000. 2-R  (dated  15  Mar  1996),  section  3. 3. 4. 3  is  the  policy 

that  directs  contractors  to  comply  with  the  C/SCSC: 

When  applicable,  the  contract  shall  require  that  any  system  used  by  the  contractor 
in  planning  and  controlling  the  performance  of  the  contract  shall  meet  the  criteria 
set  forth  in  Appendix  VI.  Nothing  in  these  criteria  is  intended  to  affect  the  basis 
on  which  costs  are  reimbursed  and  progress  payments  made,  and  nothing  herein 
shall  be  construed  as  requiring  the  use  of  any  single  system,  or  specific  method  of 
management  control  of  evaluation  of  performance.  The  government  shall  not 
require  the  contractor’s  internal  systems  to  be  changed  provided  they  satisfy  these 
criteria. 

Unless  waived  by  the  Milestone  Decision  Authority  (MDA)  or  a  designated 
representative,  compliance  with  the  C/SCSC  shall  be  required  on  significant 
contracts  and  subcontracts  within  all  acquisition  programs,  including  highly 
sensitive  classified  programs  and  major  construction  programs.  This  also  includes 
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significant  contracts  executed  for  foreign  governments  and  for  specialized 
organizations  such  as  the  Defense  Advanced  Research  Projects  Agency,  and 
significant  acquisition  effort  performed  by  Government  activities.  Significant 
contracts  include  research,  development,  test,  and  evaluation  contracts  and 
subcontracts  with  a  value  of  $70  million  or  more  or  procurement  contracts  and 
subcontracts  with  a  value  of  $300  million  or  more  (in  FY  1996  constant  dollars). 
Compliance  with  the  C/SCSC  on  contracts  and  subcontracts  below  these 
thresholds  may  be  required  when,  in  the  DoD  Component  manager’s  judgment, 
the  contract  risk  or  management  interest  requires  assurance  that  the  contractor’s 
cost  and  schedule  management  control  systems  are  acceptable.  On  contracts  that 
are  determined  to  be  not  significant  enough  for  C/SCSC  applicability,  the 
cost/schedule  status  report  (C/SSR)  (see  6.4.3)  shall  be  required  unless  excluded 
in  accordance  with  the  following  paragraph. 

Compliance  with  the  C/SCSC  shall  not  be  required  on  firm  fixed  price  contracts 
(including  firm  fixed  price  contracts  with  economic  price  adjustment  provisions), 
time  and  materials  contracts,  and  contracts  which  consist  mostly  of  level-of-effort 
work.  Exceptions  may  be  made  by  the  MDA  for  individual  contracts. 


This  excerpt  from  the  DOD  regulation  details  when  C/SCSC  compliance  is 

required.  The  latest  amendment  to  Appendix  VI  (originally  the  35  C/SCSC  were  listed  in 

Appendix  VI)  to  this  regulation  lists  the  32  EVMS  criteria  with  which  the  contractor  must 

comply  with.  It  is  important  to  understand  that  this  regulation  does  not  allow  a  program 

manager  the  flexibility  to  determine  which  contracts  must  comply  with  C/SCSC,  or  now 

EVMS.  However,  the  program  manager  has  the  power,  to  some  extent,  to  control  the 

cost  reporting  associated  with  C/SCSC  or  now  EVMS: 

The  CPR  DD  Form  2734/1, 2734/2, 2734/3, 2734/4,  and  2734/5  shall  be  used  to 
obtain  contract  cost  and  schedule  performance  information  for  use  in  making  and 
validating  program  management  decisions.  This  report  provides  early  indicators 
of  contract  cost  and  schedule  problems  and  the  effects  of  management  action 
taken  to  resolve  problems  affecting  cost  and  schedule  performance. 

1 .  CPRs  shall  be  required  on  all  contracts  that  require  compliance  with  the 
Cost/Schedule  Control  Systems  Criteria  (C/SCSC)  (see  3. 3. 4. 3  and 
Appendix  VI). 
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2.  CPRs  may  be  required  on  flexibly-priced  (for  example,  fixed-price 
incentive  or  cost  type)  contracts  that  do  not  require  compliance  with  the 
C/SCSC,  but  on  which  the  DoD  Components  require  more  data  than  is 
available  on  the  Cost/Schedule  Status  Report  (see  6.4.3).  Such 
applications  shall  not  be  used  in  lieu  of  a  valid  Cost/Schedule  Control 
Systems  Criteria  requirement.  CPR  formats,  level  of  detail,  frequency, 
and  variance  analysis  shall  be  limited  to  the  minimum  necessary  for 
effective  management  control. 

3.  CPRs  shall  not  be  required  on  firm  fixed  price  contracts  unless  unusual 
circumstances  require  cost  and  schedule  visibility. 

4.  Data  reported  on  the  CPR  shall  be  summarized  directly  from  the  same 
systems  used  for  internal  contractor  management. 

5.  The  CPR  is  subject  to  tailoring  to  require  less  data.  All  reporting 
provisions  shall  be  negotiated  and  specified  in  the  contract,  including 
reporting  frequency,  variance  analysis  requirements,  and  the  Contract 
WBS  to  be  reported.  The  CPR  is  intended  to  be  a  primary  means  of 
communication  between  the  contractor  and  the  PM  to  report  cost  and 
schedule  trends  to  date,  and  permit  assessment  of  their  likely  affect  on 
future  performance  on  the  contract.  (DOD  5000. 2-R,  6.4.2) 


This  section  describes  in  detail  when  the  CPR  must  be  used.  More  importantly, 
however,  is  the  flexibility  it  gives  the  program  manager.  For  any  given  program,  the 
CPR's  formats,  levels  of  detail,  frequency,  and  variance  analysis  are  determined  by  the 
program  manager.  The  insight  provided  by  these  two  sections  of  DOD  5000.2-R  are 
essential  in  determining  those  costs  that  can  be  avoided  by  the  program  managers,  and 
those  costs  which  can  only  be  avoided  by  changes  in  the  DOD’s  regulations. 

Now  that  we  know  what  drives  C/SCSC  compliance  and  reporting  costs,  we  can 
attempt  to  separate  these  costs  from  the  normal  operating  costs  of  a  contractor’s 
management  control  system.  In  trying  to  do  this,  several  difficulties  arise.  First,  the 
difficulty  in  measuring  the  normal  operating  costs  of  a  contractor’s  management  control 
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system  is  almost  impossible.  A  C/SCSC  study  performed  by  the  Arthur  D.  Little 
Incorporated  concluded  that  the  normal  operating  costs  are  not  available  (1983:  II-4).  A 
study  by  Lampkin  concludes  that,  “changes  in  the  contractor’s  workload,  due  to  the 
removal  of  the  mandatory  application  of  C/SCSC,  would  be  insignificant.  This 
conclusion  justifies  the  contractor’s  position  of  giving  ‘nothing’  back  to  the  government 
for  the  removal  of  the  criteria”  (1992:  30).  Even  the  recent  Earned  Value  Implementation 
Guide  acknowledges  that,  “The  cost  of  implementing  EVMS  has  defied  quantification 
due  to  the  difficulty  in  separating  the  incremental  cost  of  EVMS  from  the  normal 
management  costs  that  would  have  been  incurred  in  any  case”  (1996:  2).  In  fact,  our 
literature  review  revealed  no  studies  that  used  actual  cost  data  to  determine  the  normal 
operating  costs  of  a  contractor’s  management  control  system.  The  second  difficulty 
arises  from  the  differences  in  the  existing  management  control  systems  between 
contractors.  Some  systems  may  be  mature,  while  others  are  practically  non-existent 
(Gadeken,  1983:  1 5).  Review  of  all  known  studies  on  the  costs  of  C/SCSC  can  not 
determine  the  normal  operating  costs  of  a  contractor’s  management  control  system. 

This  chapter  will  attempt  to  identify  and  quantify  the  “generic”  costs  of  a  C/SCSC 
compliant  system.  From  this,  we  can  qualitatively  separate  those  activities  primarily 
associated  with  a  C/SCSC-compliant  system  from  those  associated  with  a  non-C/SCSC 
compliant  system  to  determine  the  cost  of  C/SCSC.  The  first  step  in  this  process  is 
breaking  out  the  cost  of  a  C/SCSC  compliant  system.  Two  main  categories  will  be  used: 
C/SCSC  system  compliance  costs,  and  C/SCSC  system  operating  costs.  C/SCSC  system 
compliance  costs  are  those  costs  associated  with  developing  and  certifying  a  C/SCSC- 
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compliant  management  system.  C/SCSC  system  operating  costs  are  those  costs 
associated  with  the  month-to-month  costs  of  the  management  system  after  it  has  been 
certified. 


C/SCSC  System  Compliance  Costs 


Three  previous  studies  broke  down  C/SCSC  system  compliance  costs  in 
determining  the  costs  of  C/SCSC. 


Table  1.  Previous  C/SCSC  Studies’  System  Compliance  Cost  Breakdown 


(Brodkorb,  1992) 

(Lampkin,  1992) 

(Gadeken,  1983) 

System  Design  Costs 

Team  Organization  and 
Start-up  Costs 

System  Assessment  Costs 

Subsystem  Development 
Costs 

Organization  Costs 

Planning  and  Budgeting 

Costs 

Accounting  Costs 

System  Documentation 

Costs 

System  Documentation 

Costs 

Analysis  Costs 

System  Implementation  and 
Operation  Costs 

System  Implementation 

Costs 

Revisions  and  Access  to 

Data  Costs 

Training  Costs 

System  Training  Costs 

Validation  Costs 

System  Evaluation  Costs 

Compliance  and 

Certification  Assistance 

Costs 

Additional  Support  Costs 

Defining  the  C/SCSC  system  compliance  costs  was  an  objective  of  the  three 
studies.  The  Brodkorb  and  Lampkin  studies  broke  these  costs  into  the  actual  steps  that 
take  place  in  developing  and  certifying  a  C/SCSC  system.  The  Gadeken  study  broke  out 
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the  C/SCSC  system  compliance  costs  by  relating  them  to  the  five  main  areas  of  the 
criteria.  The  Brodkorb  and  Lampkin  breakdowns  are  more  comparable  than  the  Gadeken. 

Based  on  these  three  studies,  this  thesis  will  define  C/SCSC  System  Compliance 
Costs  with  the  following  cost  structure: 


•  C/SCSC  System  Compliance  Costs 

•  System  Design  Costs 

•  Survey  Existing  Systems/Methods/Software  Costs 

•  Design/Integrate  Subsystems  Costs 

•  Design  reviews  Costs 

•  System  Documentation  Costs 

•  System  Description  Costs 

•  System  Procedures  Costs 

•  Training  Costs 

•  Executive  Training  Costs 

•  Intermediate  Management  Training  Costs 

•  Cost  Account  Manager  (CAM)  Training  Costs 

•  System  Implementation  Costs 

•  CPR  Preparation  Costs 

•  CA  Variance  Analysis  Costs 

•  CA  Statusing  Costs 

•  System  Evaluation  Costs 

•  Implementation  Review  Costs 

•  Readiness  Review  Costs 

•  Mock  Demonstration  Costs 

•  Demonstration  Review  Costs 


Figure  1.  C/SCSC  System  Compliance  Cost  Structure 
System  design  costs  are  those  costs  associated  with  designing  a  contractor’s 
C/SCSC-compliant  management  reporting  system.  System  documentation  costs  are  those 
costs  associated  with  the  development  and  printing  of  the  system  description  and 
procedures.  Training  costs  are  those  costs  associated  with  training  CAMs,  intermediate 
managers,  and  executive  managers  on  the  workings  of  the  contractor’s  management 
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control  system.  System  implementation  costs  consist  of  preparing  cost  account  (CA) 
plans  and  preparing  work  authorization  documents.  Additionally,  system  implementation 
costs  contain  CPR  preparation  costs,  CA  variance  analysis  costs,  and  CA  statusing  costs 
for  the  pre-system  validation  duration.  System  evaluation  costs  consist  of  the  preparation 
and  actual  system  demonstration  costs  (Brodkorb,  1992;  10-11). 

This  C/SCSC  system  compliance  cost  structure  is  primarily  based  on  the  studies 
by  Lampkin  and  Brodkorb.  In  the  study  by  Lampkin,  information  attempting  to  define  a 
generic  C/SCSC  implementation  process  was  provided  by  C/SCSC  consultants,  “based 
on  the  experiences  of  over  25  years,  working  collectively  with  more  than  750  contractors, 
and  directly  assisting  60  percent  of  the  240  contractors  that  are  currently  validated.  .  .” 

( 1 992 : 1 3).  These  consultants  also  provided  the  normal  labor  hours  associated  with  the 
implementation  tasks  (Lampkin,  1992:13).  Brodkorb  identified  five  activities  needed  to 
validate  a  C/SCSC  system.  He  attributes  these  activities  and  the  associated  person  days 
to  formulations  resulting  from  Decision  Planning  Corporation’s  20  years  of  experience  in 
assisting  contractors  in  the  development  and  installation  of  C/SCSC-compliant  systems. 
The  Brodkorb  presentation  assumes  a  generic  R&D  contract  size  of  $75M,  with  80  cost 
accounts,  40  cost  account  managers,  and  a  duration  of  three  years  (1992:6). 

Based  on  these  two  studies.  Table  2  identifies  the  person  days  associated  with 
C/SCSC  system  compliance.  The  Brodkorb  person  days  are  taken  from  his  presentation 
(1992;  7-13).  The  Lampkin  person  days  are  the  arithmetic  average  of  the  low  and  high 
values  presented  in  his  study  (1992:  17).  The  average  person  days  are  the  arithmetic 
average  of  the  Brodkorb  and  the  Lampkin  person  days. 
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Table  2.  C/SCSC  Compliance  Person  Days 


Brodkorb 

Lampkin 

IWCfll 

Person 

Days 

Person 

Days 

Person 

Days 

Person 

Days 

C/SCSC  System  Compliance  Costs 

1,197 

976 

System  Design  Costs 

240 

263 

251.5 

23.2% 

Survey  Existing 

System/Methods/Software  Costs 

20 

23 

21.5 

2.0% 

Design/Integrate  Subsystems  Costs 

190 

240 

215 

19.8% 

Design  reviews  Costs 

30 

0 

15 

1.4% 

System  Documentation  Costs 

76 

igaai 

100.75 

9.3% 

System  Description  Costs 

51 

HQQH 

51 

4.7% 

System  Procedures  Costs 

25 

N/A 

25 

2.3% 

Training  Costs 

213 

155 

184 

16.9% 

Executive  Training  Costs 

4.5 

N/A 

4.5 

0.4% 

Intermediate  Management  Training 
Costs 

126 

N/A 

126 

11.6% 

CAM  Training  Costs 

82.5 

N/A 

82.5 

7.6% 

System  Implementation  Costs 

493 

240 

366.5 

33.7% 

CPR  Preparation  Costs 

N/A 

N/A 

N/A 

CA  Variance  Analysis  Costs 

59 

N/A 

N/A 

N/A 

CA  Statusing  Costs 

60 

N/A 

N/A 

N/A 

System  Evaluation  Costs 

175 

192 

183.5 

16.9% 

Implementation  Review  Costs 

8 

14 

11 

1.0% 

Readiness  Review  Costs 

35 

42.5 

38.75 

3.6% 

Mock  Demonstration  Costs 

49 

56 

5.2% 

Demonstration  Review  Costs 

69 

86.5 

77.75 

7.2% 

Table  2  allows  us  to  quantify  the  C/SCSC  compliance  work  load  into  person  days. 
Rates  are  not  being  applied  to  these  person-days,  but  rather  percentages  are  being  used  to 
make  these  values  usable  among  different  burdened  labor  rates.  Table  2  shows  that  the 
majority,  56.9%  (23.2%  +  33.7%),  of  the  C/SCSC  system  compliance  costs  are  spent  on 
system  design  and  system  implementation  costs.  The  Lampkin  study  also  attempted  to 
quantify  these  C/SCSC  system  compliance  costs  as  a  percentage  of  the  total  contract 
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value.  The  study  cited  five  previous  studies,  all  based  on  subjective  assessments  of  costs 
from  government  and  industry  experts,  that  had  looked  at  the  C/SCSC  system  compliance 
costs.  One  of  the  previous  studies,  by  Humphreys  and  Associates,  broke  the  cost 
percentages  into  two  categories:  recurring  and  non-recurring.  The  recurring  costs  are 
those  C/SCSC  system  compliance  costs  related  to  a  previously  validated  system.  The 
non-recurring  costs  are  those  C/SCSC  system  compliance  costs  associated  with  a 
contractor  going  through  the  validation  process  for  the  first  time  ( Incidentally,  the 
C/SCSC  system  compliance  person  days  shown  in  Table  2  are  of  the  non-recurring  type). 
Lampkin  also  stated  that  this  was  the  only  previous  study  to  differentiate  between  the  two 
types  of  costs  (1992:  34-37).  Table  3  summarizes  Lampkin’s  conclusions: 


Table  3.  C/SCSC  System  Compliance  Costs  (%  of  Total  Contract  Value) 


Study  (Year) 

Cost  Range 

Basis 

Recurring 

MITRE 

0.1  -0.2% 

Software  Acq  Model 

N/A 

N/A 

SPARROW 

0.6-  1.0% 

Cost  Est.  Model 

N/A 

N/A 

KOUTS 

0.5  -  5.0%* 

Industry  Response 

N/A 

N/A 

DoDIG 

5.0% 

DoD  Response 

N/A 

N/A 

HUMPHREYS 

0.5%  -  4.0% 

Consultant  Response 

0.5%  -  1.5% 

Lampkin  Avg  Range 

0.4% 

1.63% 

Lampkin  Mean  of 
Averages 

1.69% 

N/A 

N/A 

*Note:  3%  is  the  average. 


Table  3  provides  a  summary  of  past  C/SCSC  system  compliance  cost  studies. 
However,  no  assumptions  or  details  about  these  studies  are  given,  thus  the  credibility  of 


these  findings  is  somewhat  questionable. 

The  final  consideration  in  the  C/SCSC  system  compliance  costs  is  the  government 
costs.  The  government  will  incur  costs  both  for  personnel  assigned  to  review  a 


contractor’s  management  control  system  and  for  travel. 


20 


C/SCSC  System  Operating  Costs 


Unlike  C/SCSC  system  compliance  costs,  C/SCSC  system  operating  costs  are 
extremely  dependent  on  the  program  manager’s  contractual  cost  reporting  requirements. 
The  program  manager  controls  the  frequency,  WBS  level  and  format  of  the  cost 
reporting,  and  the  detail  and  thresholds  of  variance  analysis  required.  These  factors  can 
cause  wide  fluctuations  in  the  C/SCSC  system  operating  costs. 

Based  on  the  1992  Brodkorb  presentation,  the  following  relationship  equations 
will  be  used  to  estimate  the  cost  of  the  C/SCSC  system  operating  costs. 


Table  4.  C/SCSC  System  Operating  Hours 


Activity 

Estimating  Relationship  Equations 

CPR  Preparation  Hours 

#  of  Post  Validation  CPRs  cycles  *  40  hrs/CPR 

CA  Variance  Analysis  Hours 

#  of  Post  Validation  CPR  cycles  *  (1/3)  *  #  of 

CAs  *  3hrs/CA 

CA  Plan  Statusing  Hours 

#  of  Post  Validation  CPR  cycles  *  #  of  CAs  * 

Ihr/CA 

Total  C/SCSC  System  Operating  Hours 

[40  +  (2  *  #  of  CAs)]  *  (#  of  Post  Validation 

CPR  cycles) 

To  illustrate  this,  let’s  assume  a  3  year  contract,  with  80  cost  accounts,  monthly 
CPR  reporting,  and  a  validation  duration  of  9  months.  The  preparation  of  the  CPR  for  the 
remaining  27  months  takes  1,080  hours  (27  *  40),  or  135  person  days  (1,080  /  8).  The 
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CA  variance  analysis  takes  2,160  hours  (27  *  1/3  *  80  *  3),  or  270  person  days  (2,160  / 
8).  The  CA  plan  statusing  also  takes  2,160  hours  (27  *  80  *  1),  or  270  person  days  (2,160 
/  8).  From  this,  the  total  C/SCSC  system  operating  hours  are  5,400  (1,080  +  2,160  + 
2,160),  or  675  person  days  (5,400  /  8). 

In  addition  to  the  contractor’s  costs,  the  government  also  has  C/SCSC  system 
operating  costs.  Government  person  hours  are  used  to  analyze  CPRs,  prepare  EACs,  and 
implement  baseline  changes. 


Conclusion 

This  chapter  began  by  attempting  “. . .  to  separate  the  normal  operating  costs  of  a 
contractor's  management  control  system  from  the  total  cost  of  operating  a  C/SCSC 
compliant  system  (Lampkin,  1992:  3).  We  have  identified  the  total  costs  of  operating  a 
C/SCSC  compliant  system,  and  will  now  qualitatively  separate  the  ‘normal’  operating 
costs. 

C/SCSC  compliance  eosts  were  separated  into  five  categories:  1)  system  design 
costs.  2)  system  documentation  costs,  3)  training  costs,  4)  system  implementation  costs, 
and  5)  system  evaluation  costs.  System  design  costs  are  necessary  for  both  the  C/SCSC 
compliant  systems  as  well  as  the  ‘normal’  systems.  It  is  very  obvious  that  a  commercial 
management  control  system  must  first  be  designed  before  it  can  be  used. 

System  documentation  costs  are  not  clearly  a  needed  product  of  a  ‘normal’ 
system.  Added  to  this  are  the  32  criteria  (35  under  C/SCSC)  which  the  system  must 
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meet.  In  a  1992  study  by  Lampkin,  six  out  of  eight  contractors  had  to  develop  system 
descriptions  to  comply  with  C/SCSC.  Also,  it  was  determined  that  fifty  percent  of  these 
costs  are  ‘CS  only’  costs.  Additionally,  all  eight  contractors  responded  that  they  had  to 
expand  their  written  procedures  to  comply  with  C/SCSC.  With  this  in  mind,  it  is  a 
reasonable  assessment  that  the  system  documentation  for  a  ‘normal’  system  would  be 
half  that  of  a  C/SCSC  compliant  system. 

System  training  costs  are  clearly  needed  for  both  a  ‘normal’  and  a  C/SCSC- 
compliant  management  control  system.  A  contractor  needs  to  train  the  users  of  their 
system  whether  it  is  C/SCSC  compliant  or  not. 

System  implementation  costs  are  those  operating  costs  incurred  before  the 
contractor’s  system  is  validated.  In  essence,  they  are  a  means  for  testing  the  management 
control  system.  This  would  have  to  be  done  with  a  ‘normal’  system,  however,  the 
amount  of  reporting  and  variance  analysis  performed  as  a  result  of  C/SCSC  has  been 
excessive  in  the  past.  The  1992  study  by  Lampkin  found  that  system  implementation 
activities  such  as  performance  reporting  procedure  needed  to  be  developed  by  the 
majority  of  the  contractors  surveyed.  The  ‘CS  only’  costs  ranged  from  30%  to  70%. 

This  reporting  and  variance  analysis  can  be  reduced  by  the  government  program  manager, 
however,  we  conclude  that  the  ‘normal’  system  implementation  costs  will  range  from 
30%  to  70%  of  C/SCSC-compliant  system  implementation  costs. 

Finally,  it  is  understood  that  the  majority  of  system  evaluation  costs  are  only 
associated  with  C/SCSC-compliant  systems.  In  the  1992  study  by  Lampkin,  95%  of  the 
system  evaluation  costs  were  defined  as  ‘CS  only’  costs.  Henceforth,  ‘normal’  system 
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evaluation  costs  are  estimated  as  5%  of  C/SCSC-compliant  system  implementation  costs. 
The  above  analysis  leads  us  to  conclude  the  following: 


Table  5.  Marginal  C/SCSC  Compliance  Costs 


C/SCS 

Compliant 

Costs 

‘Normal’ 

System 

Compliance 

Costs 

Marginal  Cost 
of  C/SCSC 
Compliance 

Person  Days 

Person  Days 

Person  Days 

System  Compliance  Costs 

1,086 

605-752 

334-481 

System  Design  Costs 

251.5 

251.5 

0 

System  Documentation 
Costs 

100.75 

50.38 

50.38 

Training  Costs 

184 

184 

0 

System  Implementation 
Costs 

366.5 

109.95-256.55 

109.95-256.55 

System  Evaluation  Costs 

183.5 

9.18 

174.32 

From  this  table,  we  conclude  that  the  marginal  C/SCSC  compliance  costs  are  31- 
44%  (Marginal  Cost  divided  by  C/SCSC-compliant  costs)  of  the  total  C/SCSC 
compliance  costs. 

As  acquisition  reform  continues  to  model  government  practices  after  commercial 
practices,  the  difference  between  a  C/SCSC-compliant  system  and  a  ‘normal’  system  will 
hopefully  decrease.  Changes  in  C/SCSC  reporting  requirements  and  system  acceptance 
procedures  can  decrease  the  marginal  cost. 

‘Normal’  system  operating  costs  are  concluded  as  50%  of  C/SCSC.  The  1992 
study  by  Lampkin  determined  that  fifty  percent  of  these  operating  costs  are  ‘CS  only.’  It 
is  evident  that  there  are  excessive  reporting  requirements,  “. . .  all  contractors  subject  to 
C/SCSC  agree  that,  as  currently  required  by  the  DoD,  cost/schedule  reporting  is  too 
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detailed,  repetitive,  and  voluminous  to  be  used  effectively  as  a  management  tool  by  either 
the  government  or  industry, . . (Coopers  &  Lybrand  and  TASC,  1994:  22).  One 
initiative  to  decrease  criteria-compliant  system  operating  costs  is  summary  level  variance 
analysis.  This  initiative  combines  cost  accounts  in  an  effort  to  reduce  the  volume  of 
variance  analysis  and  was  tested  at  some  defense  contractors.  IBM  Federal  Systems  cites 
a  45%  reduction  in  variance  analysis  volume  attributed  to  summary  level  variance 
analysis.  McDonnell  Douglas  Aerospace  claims  a  20%-40%  cost  reduction  due  to 
summary  level  variance  analysis  (Pakiz,  1993:  1-7).  These  cost  savings  realized  from 
this  commercial  practice  are  just  an  example  of  the  marginal  system  operating  costs  that 
can  be  reduced. 

It  is  important  to  compare  these  conclusions  to  the  findings  of  the  1994  Coopers 
&  Lybrand/TASC  report.  This  report  concluded  that  DOD  regulatory  compliance  costs 
are  1 8%  of  the  ‘value  added’  costs  (47).  ‘Value  added’  costs  are  defined  as  total 
contractor  costs  less  material  costs,  and  are  approximately  60%  of  total  costs  (4).  This 
report  also  concluded  that  C/SCSC  regulatory  requirements  account  for  0.9%  of  the  total 
‘value  added’  cost  (18).  For  example,  on  a  $100M  contract,  $60M  are  identified  as 
‘value  added’  costs.  Of  this  $60M,  18%  ($10.8M)  are  a  result  of  regulatory  compliance. 
The  C/SCSC  compliance  costs  in  this  example  would  be  0.9%  of  $60M  or  $0.54M.  This 
cost  represents  0.54%  of  the  total  contract  value  ($100M)  and  is  consistent  with  the 
findings  in  Table  3.  While  the  C/SCSC  regulatory  compliance  costs  do  not  seem  very 
high,  it  ranked  third  in  the  report’s  top  ten  cost  drivers.  This  report  drove  the  need  for  the 
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establishment  of  the  "...  Regulatory  Cost  Premium  Working  Group  to  coordinate  DOD- 

wide  efforts  to  address  cost  drivers”  (GAO,  1996:  2). 

With  all  of  the  attention  concentrating  on  the  costs  of  C/SCSC  regulatory 

compliance,  the  benefits  of  compliance  were  not  addressed.  Coopers  &  Lybrand/TASC 

stated  that,  “  . . .  some  claim  that  DOD  receives  substantial  benefits  from  its  regulatory 

activities.  The  Project  Team  did  not  attempt  to  validate  the  existence  of  such  benefits  or 

quantify  their  value.  In  other  words,  we  looked  only  at  the  ‘cost’  portion  of  the 

cost/benefit  ratio”  (1994:  3).  The  GAO  report  addressing  the  Coopers  &  Lybrand/TASC 

report,  and  the  results  from  the  Regulatory  Cost  Premium  Working  Group  stated  that,  “  . . 

.  sa\  ings  resulting  from  current  DOD  initiatives  may  be  significantly  less  than  the  1 8- 

percent  cost  premium  identified  by  Coopers  and  Lybrand”  (1996:  3).  The  GAO  report 

then  addresses  why  the  actual  savings  are  significantly  less. 

One  explanation  for  the  disparity  between  Coopers  and  Lybrand’s  1 8-percent  cost 
premium  and  the  reinvention  laboratory  results  is  that  Coopers  and  Lybrand  did 
not  attempt  to  assess  the  benefits  resulting  from  the  cost  drivers  it  identified. 
Rather,  the  Coopers  and  Lybrand  study  analyzed  only  the  cost  impact  of  DOD’ s 
regulations  and  oversight  requirements  on  contractors.  DOD’s  Reducing 
Oversight  Costs  Reinvention  Laboratory  identified  a  significantly  lower  cost 
savings  potential  because  some  of  the  cost  drivers  made  good  business  sense.  In 
addition,  contractors  would  have  similar  self-imposed  requirements  in  the  absence 
of  some  of  DOD’s  regulatory  requirements.  (1996:  5) 

To  make  a  decision  on  the  need  for  C/SCSC,  it  is  essential  to  look  at  both  the 

costs  and  benefits.  This  chapter  detailed  the  costs,  while  Chapter  III  will  outline  and 

discuss  the  benefits  of  C/SCSC. 
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III.  Benefits  of  C/SCSC 


Overview 

What  are  the  benefits  that  result  from  using  a  C/SCSC  compliant  system?  During 
our  research  we  found  much  literature  on  the  subject  that  lends  general,  as  well  as  very 
specific,  support  for  the  criteria  (GAO,  1997;  Coopers  &  Lybrand  and  TASC,  1994; 
DoD/NSlA,  1991;  Little,  1984;  Little,  1983;  NSIA,  1980).  We  will  examine  these 
studies,  and  other  literature,  related  to  the  benefits  of  C/SCSC  (or  EVMS)  in  more  detail 
later  in  this  chapter.  But  first,  we  will  review  the  actual  criteria  as  spelled  out  in 
Appendix  VI  of  DoD  5000.2-R.  Next,  we  will  explain  how  earned  value  fits  into  the 
picture.  Finally,  we  will  show  how  the  criteria  may  result  in  many  benefits  to  the  DoD 
and  to  government  contractors. 


The  Specific  Criteria 

Most  of  the  material  we  draw  upon  in  this  chapter  is  related  specifically  to 
C/SCSC.  Therefore,  we  will  address  the  benefits  of  C/SCSC  first.  Later  on,  we  will 
point  out  the  differences  between  the  new  EVMS  criteria  and  the  former  C/SCSC  criteria 
they  replaced.  Then,  we  will  describe  how  the  new  criteria  affect  these  same  costs  and 
benefits  that  we  present  in  this  paper.  Table  6  provides  a  list  of  the  35  C/SCSC  criteria. 
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Table  6.  C/SCSC  Criteria  (DoD  5000.2-R  Appendix  VI) 


Organization 

1. 

Define  all  authorized  work  and  related  resources  to  meet  the  requirements  of  the 
contract,  using  the  framework  of  the  contract  work  breakdown  structure. 

2. 

Identify  the  internal  organizational  elements  and  the  major  subcontractors 
responsible  for  accomplishing  the  authorized  work. 

3. 

Provide  for  the  integration  of  the  contractor's  planning,  scheduling,  budgeting, 
work  authorization,  and  cost  accumulation  systems  with  each  other,  the  contract 
work  breakdown  structure,  and  the  organizational  structure. 

4. 

Identify  the  managerial  positions  responsible  for  controlling  overhead  (indirect 
costs). 

5. 

Provide  for  integration  of  the  contract  work  breakdown  structure  with  the 
contractor's  functional  organizational  structure  in  a  manner  that  permits  cost  and 
schedule  performance  measurement  for  contract  work  breakdown  structure  and 
organizational  elements. 

Planning  &  budgeting 

6. 

Schedule  the  authorized  work  in  a  manner  that  describes  the  sequence  of  work 
and  identifies  the  significant  task  interdependencies  required  to  meet  the 
development,  production,  and  delivery  requirements  of  the  contract. 

7. 

Identify  physical  products,  milestones,  technical  performance  goals,  or  other 
indicators  that  will  be  used  to  measure  output. 

8. 

Establish  and  maintain  a  time-phased  budget  baseline  at  the  cost  account  level 
against  which  contract  performance  can  be  measured.  Initial  budgets  established 
for  this  purpose  will  be  based  on  the  negotiated  target  cost.  Any  other  amount 
used  for  performance  measurement  purposes  must  be  formally  recognized  by 
both  the  contractor  and  the  government. 

9. 

Establish  budgets  for  all  authorized  work  with  separate  identification  of  cost 
elements  (labor,  material,  etc.). 

10. 

To  the  extent  the  authorized  work  can  be  identified  in  discrete,  short-  span  work 
packages,  establish  budgets  for  this  work  in  terms  of  dollars,  hours,  or  other 
measurable  units.  Where  the  entire  cost  account  cannot  be  subdivided  into 
detailed  work  packages,  identify  the  far-term  effort  in  larger  planning  packages 
for  budget  and  scheduling  purposes. 

11. 

Provide  that  the  sum  of  all  work  package  budgets  plus  planning  packages  within 
a  cost  account  equals  the  cost  account  budget. 

12. 

Identify  relationships  of  budgets  or  standards  in  underlying  work  authorization 
systems  to  budgets  for  work  packages. 

13. 

Identify  and  control  level  of  effort  activity  by  time-phased  budgets  established 
for  this  purpose.  Only  that  effort  which  cannot  be  identified  as  discrete  or  as 
apportioned  effort  will  be  classed  as  level  of  effort. 
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Establish  overhead  budgets  for  the  total  costs  of  each  significant  organizational 
component  whose  expenses  will  become  indirect  costs.  Reflect  in  the  contract 
budgets,  at  the  appropriate  level,  the  amounts  in  overhead  pools  that  will  be 
allocated  to  the  contract  as  indirect  costs. 

Identify  management  reserves  and  undistributed  budget. 

Provide  that  the  contract  target  cost  plus  the  estimated  cost  of  authorized  but 
unpriced  work  is  reconciled  with  the  sum  of  all  internal  contract  budgets  and 
management  reserves. 

Accounting 

Record  direct  costs  on  an  applied  or  other  acceptable  basis  in  a  formal  system 
that  is  controlled  by  the  general  books  of  account. 

Summarize  direct  costs  from  cost  accounts  into  the  work  breakdown  structure 
without  allocation  of  a  single  cost  account  to  two  or  more  work  breakdown 
structure  elements. 

Summarize  direct  costs  from  the  cost  accounts  into  the  contractor's  functional 
organizational  elements  without  allocation  of  a  single  cost  account  to  two  or 
more  organizational  elements. 

Record  all  indirect  costs  that  will  be  allocated  to  the  contract. 

Identify  the  basis  for  allocating  the  cost  of  apportioned  effort. 

Identify  unit  costs,  equivalent  unit  costs,  or  lot  costs,  as  applicable. 

The  contractor's  material  accounting  system  will  provide  for;  a.  Accurate  cost 
accumulation  and  assignment  of  costs  to  cost  accounts  in  a  manner  consistent 
with  the  budgets  using  recognized,  acceptable  costing  techniques. 
Determination  of  price  variances  by  comparing  planned  vs  actual  commitments. 
Cost  performance  measurement  at  the  point  in  time  most  suitable  for  the 
category  of  material  involved,  but  no  earlier  than  the  time  of  actual  receipt  of 
material. 

Determination  of  cost  variances  attributable  to  the  excess  usage  of  material. 
Determination  of  unit  or  lot  costs  when  applicable. 

Full  accountability  for  all  material  purchased  for  the  contract,  including  the 
residual  inventory. 

Analysis 

Identify  at  the  cost  account  level  on  a  monthly  basis  using  data  fi'om,  or 
reconcilable  with,  the  accounting  system: 

a.  Budgeted  cost  of  work  scheduled  and  budgeted  cost  of  work  performed. 

b.  Budgeted  cost  of  work  performed  and  applied  (actual  where  appropriate) 
direct  costs  for  the  same  work. 

c.  Variances  resulting  from  the  above  comparisons  classified  in  terms  of  labor, 
material,  or  other  appropriate  elements  together  with  the  reasons  for  significant 
variances. 
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25. 

Identify  on  a  monthly  basis,  in  detail  needed  by  management  for  effective 
control,  budgeted  indirect  costs,  actual  indirect  costs,  and  variances  along  with 
the  reasons. 

26. 

Summarize  the  data  elements  and  associated  variances  listed  in  [24]  and  [25] 
above,  through  the  contractor  organization  and  work  breakdown  structure  to  the 
reporting  level  specified  in  the  contract. 

27. 

Identify  significant  differences  on  a  monthly  basis  between  planned  and  actual 
schedule  accomplishment  and  the  reasons. 

28. 

Identify  managerial  actions  taken  as  a  result  of  criteria  items  [24]  through  [27] 
above. 

29. 

Based  on  performance  to  date,  on  commitment  values  for  material,  and  on 
estimates  of  future  conditions,  develop  revised  estimates  of  cost  at  completion 
for  work  breakdown  structure  elements  identified  in  the  contract  and  compare 
these  with  the  contract  budget  base  and  the  latest  statement  of  funds 
requirements  reported  to  the  government. 

Revisions  and  access  to  data 

30. 

Incorporate  contractual  changes  in  a  timely  manner,  recording  the  effects  of 
such  changes  in  budgets  and  schedules.  In  the  directed  effort  before  negotiation 
of  a  change,  base  such  revisions  on  the  amount  estimated  and  budgeted  to  the 
functional  organization. 

31. 

Reconcile  original  budgets  for  those  elements  of  the  work  breakdown  structure 
identified  as  priced  line  items  in  the  contract,  and  for  those  elements  at  the 
lowest  level  of  the  DOD  program  work  breakdown  structure,  with  current 
performance  measurement 

32. 

Prohibit  retroactive  changes  to  records  pertaining  to  work  performed  that  will 
change  previously  reported  amounts  for  direct  costs,  indirect  costs,  or  budgets, 
except  for  correction  of  errors  and  routine  accounting  adjustments. 

33. 

Prevent  revisions  to  the  contract  budget  base  except  for  government-  directed 
changes  to  contractual  effort. 

34. 

Document  internally,  changes  to  the  performance  measurement  baseline  and 
notify  the  procuring  activity  expeditiously  through  prescribed  procedures. 

35. 

Provide  the  contracting  officer  and  the  contracting  officer's  authorized 
representatives  with  access  to  the  information  and  supporting  documents 
necessary  to  demonstrate  compliance  with  the  cost/schedule  control  system 
criteria. 
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Criteria  Benefits  versus  Earned  Value  Benefits 


Before  delving  into  the  details  of  the  benefits  we  should  make  a  distinction 
between  earned  value  benefits  and  benefits  of  the  criteria.  But  first,  we  will  briefly 
review  the  concept  of  earned  value. 

Earned  value  is  a  management  technique  that  relates  resource  planning 
to  schedules  and  to  technical  performance  requirements.  Without 
earned  value,  one  can  only  compare  how  much  has  been  spent  with 
what  was  planned  to  be  spent  ("spend  plan"  management),  with  no 
objective  indication  of  how  much  of  the  planned  work  actually  was 
accomplished  for  those  dollars  spent.  Cost/Schedule  Control  Systems 
Criteria  (C/SC  SC)  require  the  contractor  to  plan,  budget,  and  schedule 
all  authorized  effort  in  time-phased  "platmed  value"  increments 
constituting  a  "performance  measurement  baseline."  As  work  is 
accomplished,  it  is  "earned"  on  the  same  basis  it  was  planned,  in 
dollars  or  other  measurable  units,  such  as  labor  hours.  Planned  value 
compared  with  earned  value  thus  measures  the  dollar  volume  of  work 
planned  versus  the  equivalent  dollar  volume  of  work  accomplished. 
(Deskbook,2.B.2.1) 

A  C/SCSC  compliant  system  has  the  ability  to  produce  the  data  needed  to  perform 
earned  value  analyses.  Our  research  indicates  many  benefits  can  be  obtained  through  the 
use  of  earned  value  analysis  techniques.  However,  earned  value  techniques  are  only  a 
subset  of  the  benefits  related  to  C/SCSC.  Indeed,  there  are  many  other  benefits  resulting 
from  the  use  of  the  criteria  but  unrelated  to  earned  value  analysis  techniques.  Clearly, 
earned  value  analysis  can  be  performed  without  being  required  by  the  criteria.  However, 
in  the  absence  of  certain  criteria,  many  of  these  benefits  may  be  less  achievable.  Without 
the  discipline  required  by  the  criteria,  how  can  we  ensure  the  data  is  accurate?  Without 
accurate  data,  how  valid  will  the  earned  value  analyses  be?  Thus,  we  assert  that  any 


31 


benefits  derived  through  earned  value  techniques  are  ultimately  benefits  attributable  to 
the  criteria. 


Ten  Benefits  of  Earned  Value 


Fleming  and  Koppelman  recently  published  a  book  regarding  earned  value  with 
one  chapter  explicitly  describing  the  benefits  of  C/SCSC  (Fleming  and  Koppelman,  1 996; 
21-29).  In  this  chapter,  they  provide  a  list  of  ten  of  the  most  important  contributions 
resulting  from  the  employment  of  C/SCSC.  This  list  is  based  on  their  review  of  the 
existing  body  of  knowledge  on  the  subject  and  is  summarized  in  Table  7  below. 


Table  7.  Fleming  and  Koppelman’s  Benefits  of  C/SCSC 


1. 

Employment  of  a  single  management  control  system  which  provides  accurate, 
consistent,  reliable,  and  timely  data  for  use  by  all  levels  of  management. 

2. 

A  management  approach  that  integrates  cost,  schedule  and  technical  parameters 
allowing  for  the  continuous  measurement  of  integrated  performance. 

3. 

Documented  empirical  database  collected  from  [thousands]  of  DoD  contracts 
reflecting  a  consistent  and  predictable  performance  history. 

4. 

Utility  and  stability  of  the  cumulative  Cost  Performance  Index  (CPI  cum)  I® 
continuously  monitor  performance  trends  of  a  project. 

5. 

Utility  of  the  Schedule  Performance  Index  (SPI)  to  monitor  and  quantify  the 
value  of  the  work  scheduled  and  to  compare  it  against  work  performed. 

6. 

Utility  of  the  CPIcum  to  statistically  forecast  low  end  range  of  completion  costs. 

7. 

Utility  of  the  CPIcum  combined  with  SPI  to  forecast  most  likely  completion  cost. 

8. 

Utility  of  the  To-Complete  Performance  Index  (TCPI)  to  monitor  remaining  tasks 
against  a  specific  management  financial  goal. 

9. 

Utility  of  a  weekly  (or  periodic)  CPI  to  measure  actual  performance  against  a  pre¬ 
set  standard. 

10. 

The  use  of  Management  by  Exception  principles  to  focus  management’s  attention 
on  significant  exceptions  to  the  authorized  plan. 

32 


These  ten  areas  will  serve  as  the  major  backbone  of  this  chapter  regarding  the  benefits  of 
C/SCSC.  The  rest  of  the  chapter  will  expound  upon  Fleming  and  Koppelman’s  work  by 
linking  it  with  other  literature  related  to  benefits  of  the  criteria. 


The  Ten  Benefits  Described 

Benefit  One.  The  first  benefit  highlighted  by  Fleming  and  Koppelman  is  derived  fi'om 
the  employment  of  a  single  management  control  system.  In  the  past,  many  problems 
resulted  from  contractors  managing  with  one  system  and  reporting  from  another.  The 
first  five  criteria  under  C/SCSC  or  EVMS  deal  with  organization  and  are  closely  related 
to  this  benefit.  These  criteria  permit  only  one  work  breakdown  structure  per  contract. 
Therefore,  contractors  are  forced  to  manage  and  report  from  the  same  system.  The 
motivation  to  require  a  singular  system  came  about  because  the  contractor’s 


. . .  project  manager,  senior  corporate  management,  chief 
financial  officer,  and  others,  often  had  their  own  data, 
which  frequently  did  not  match  other  data.  Obviously,  this 
condition  produced  disastrous  performance  results  ...  a 
single  management  control  system  provides  accurate, 
consistent,  reliable,  and  timely  data  that  management  at  all 
levels  can  use  to  monitor  performance  throughout  the  life 
cycle  of  a  project.  (Fleming  and  Koppelman,  1996;  22) 

Although  the  new  EVMS  criteria  do  not  specify  the  exact  format  of  the  work 
breakdown  structure,  the  requirement  for  the  contractor  to  develop  a  WBS  is  listed  in 
section  4.4.2  of  DoD  5000.2-R.  Therefore,  there  is  very  little  difference  between  the  five 
new  EVMS  organizational  criteria  and  the  old  C/SCSC  organizational  criteria.  Under 


33 


EVMS.  the  contractor  is  still  required  to  perform  the  same  basic  steps  as  before  under  the 
C/SCSC  organizational  criteria.  The  contractor  is  still  required  to:  define  the  authorized 
work  elements  for  the  program;  identify  the  program  organizational  structure;  define  the 
organizational  elements  in  which  work  will  be  planned  and  controlled;  provide  for  the 
integration  of  the  company’s  planning,  scheduling,  budgeting,  work  authorization,  and 
cost  accumulation  processes  with  each  other;  identify  the  company  organization  or 
function  responsible  for  controlling  overhead;  and  provide  for  integration  of  the  program 
work  breakdown  structure  and  the  program  organizational  structure  in  a  manner  that 
permits  cost  and  schedule  performance  measurement  by  elements  of  either  or  both 
structures  as  needed. 

In  the  Preface  to  the  C/SCSC  White  Paper,  Abba  explains  that  “C/SCSC  provide 
the  best  tool  available  to  assure  contractors  have  and  use  adequate  cost  and  schedule 
management  control  systems...  a  C/SCSC  compliant  system  can  ensure  that  valid  cost 
and  schedule  performance  data  are  generated,  easing  the  manager’s  task  in  making  the 
correct  decision”  (1986:  viii). 

In  the  past,  contractors  were  required  to  prove  to  the  government  that  their 
management  control  systems  were  in  compliance  with  C/SCSC  requirements  through  a 
formal  validation  process.  Presutti,  regarding  a  validated  system,  states  “this  approval 
gives  any  user  some  confidence  in  the  information  that  the  system  can  provide”  (1993: 
53).  Not  only  will  reliable  data  foster  good  decision  making;  greater  confidence  in  that 
data  can  multiply  the  effect.  In  other  words,  managers  should  be  able  to  make  better 
decisions  if  they  have  more  confidence  in  the  information  provided  by  their  system. 
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Benefit  Two.  Another  benefit  gained  through  the  use  of  C/SCSC  is  “a  management 


approach  that  integrates  the  technical  scope  of  work,  the  schedules,  and  the  costs, 
allowing  for  the  continuous  measurement  of  integrated  performance  throughout  the  life 
cycle  of  the  project  or  a  production  effort”  (Fleming  and  Koppelman,  1996:  23).  The 
third  criterion,  under  both  C/SCSC  and  EVMS,  requires  the  contractor  to  integrate  their 
planning,  scheduling,  budgeting,  work  authorization  and  cost  accumulation  systems  with 
each  other. 

Instead  of  projects  being  defined  and  measured  by  parochial  interests  in  a  non- 
integrated  manner  as  was  frequently  the  case  in  the  past,  now  performance  measurement 
could  take  place  in  a  much  more  highly  integrated  manner  (Fleming  and  Koppelman, 
1996:  23).  This  new  integrated  approach  served  to  help  various  functional  personnel  to 
“speak  a  common  project  language  for  the  first  time”  and  allowed  the  various  functional 
metrics  to  be  much  easier  to  relate  to  each  other  (Fleming  and  Koppelman,  1996:  23). 

Responses  to  the  National  Security  Industrial  Association  (NSIA)  survey  show 
that  when  asked  what  measurements  are  visible  in  their  labor  work  package  system 
beside  budget,  the  respondents  indicated  that  earned  value  is  used  more  than  any  other 
measurement  listed,  including  “underlying  standards/goals  [of  the  organization]”  (NSIA, 
1980:  III- 15).  The  purpose  of  the  NSIA  study  was  to  provide  industry  with  a  data  base  to 
aid  contractors  recently  exposed  to  Performance  Measurement  System  Requirements  as 
well  as  contractors  with  previously  accepted  systems.  Another  objective  of  this  project 
was  to  provide  data  from  which  future  improvements  could  be  made  in  requirements  and 
implementation  practices.  Seventy-four  companies  responded  to  this  questionnaire. 
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Benefit  Three.  Another  benefit  resulting  from  the  use  of  C/SCSC  is  the  documented 


empirical  data  compiled  over  the  years  from  thousands  of  DOD  contracts.  The  first 
criterion,  under  C/SCSC  and  EVMS,  requires  contractors  to  define  the  authorized  work 
elements.  Under  C/SCSC,  this  criterion  requires  preparation  of  a  work  breakdown 
structure  consistent  with  governmental  guidelines.  Development  of  a  WBS  in  this 
manner  allows  the  government  to  be  able  to  compare  that  program  with  others  in  its 
historical  database  (Fleming,  1992:  79).  Under  EVMS,  however,  the  requirement  for  a 
WBS  is  not  specifically  stated.  Instead,  the  WBS  requirement  is  provided  in  DoD 
5000. 2-R  with  guidelines  for  its  development  found  in  MIL-HDBK-881. 

The  analysis  of  contracts  employing  earned  value  has  provided  some  important 
insights  about  government  contracts.  After  a  project  is  about  20%  complete,  history  has 
shown  there  is  very  little  chance  that  any  overruns  can  be  recovered,  and  will  usually 
worsen  (Christensen,  1993:  15).  Even  with  this  knowledge,  it  is  still  very  beneficial  to  be 
able  to  predict,  with  some  degree  of  confidence,  the  final  cost  and  schedule  early  in  a 
project. 

The  A- 12  acquisition  program  highlights  the  benefits  of  having  such  a  database. 

In  an  interview  with  David  Christensen,  Ph.D.,  he  stated  “because  we  had  a  reliable, 
comprehensive  record  of  performance  on  other,  similar  projects,  we  could  better  evaluate 
the  progress  on  the  A- 12  program”  (Christensen,  1997).  Figure  2  depicts  how  the 
performance  of  the  A- 12  compared  with  other  programs  (Abba,  1991 :  13). 
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Figure  2.  A-12  Performance  Comparison 

Christensen  stated  that  this  graph  was  presented  to  Secretary  of  Defense,  Richard  Cheney, 
and  was  regarded  as  compelling  evidence  that  the  A-12  program  was  out  of  control 
(Christensen,  1997). 

Benefit  Four.  Another  major  benefit  gleaned  from  the  use  of  C/SC  SC  stems  from  the 
utility  and  the  stability  of  the  cumulative  Cost  Performance  Index  (CPIcum)-  The  seven 
accounting  criteria  (six  under  EVMS)  establish  the  foundation  for  this  benefit.  These 
criteria  do  not  require  a  specific  accounting  system,  but  they  do  require  proper  interfaces 
be  established  between  the  accounting  system  and  the  performance  measurement  system. 
This  “ensures  compatibility  of  actual  costs  and  budgeted  costs  in  the  performance 
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reports”  (Presutti,  1993:  52).  Of  course,  this  compatibility  is  fundamental  to  valid  and 
accurate  earned  value  analyses. 

Again,  using  cumulative  contract  performance  data,  management  is  able  to  assess 
the  performance  to  date  and  to  predict  the  final  performance  results  quite  well  at  a  very 
early  point.  In  fact,  “based  on  data  from  the  Defense  Acquisition  Executive  Summary 
database,  results  indicate  that  the  cumulative  CPI  is  stable  from  the  20  percent 
completion  point  regardless  of  contract  type,  program,  or  service”  (Christensen  and 
Heise,  1993:  7). 

Benefit  Five.  Another  area  of  benefit  regarding  C/SCSC  is  gained  through  the  use  of 
the  Schedule  Performance  Index  (SPI).  The  SPI  can  be  used  to  monitor  and  quantify  the 
value  of  the  work  scheduled  and  to  compare  it  against  the  value  of  work  physically 
performed.  Criteria  6,  7  and  27,  under  C/SCSC  (criteria  6,  7,  and  25  under  EVMS),  deal 
with  the  need  to  have  a  formal  scheduling  process.  Regarding  these  criteria,  Fleming  and 
Koppelman  ask  the  rhetorical  question:  ”  are  these  three  criteria  unique  to  an  earned  value 
approach?”  Their  response  is:  “Absolutely  not.  These  three  criteria  lay  down 
fundamental  scheduling  principles  which  would  apply  to  any  project  anywhere  in  the 
world”  (1996:  48). 

Contractors  in  compliance  with  C/SCSC  can  easily  compute  this  index  which 
allows  a  quick  determination  of  how  far  ahead  or  behind  the  planned  schedule  they  are. 
The  SPI  can  be  used  in  conjunction  with  the  Critical  Path  Method  (CPM)  to  help  manage 
the  project  schedule  more  effectively. 
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Benefit  Six.  Another  benefit  of  C/SC  SC  is  related  to  the  utility  of  the  cumulative  Cost 
Performance  Index  (CPIcum)-  Criterion  29,  under  C/SCSC  (  criterion  27  under  EVMS), 
requires  the  contractor  to  develop  a  revised  estimate  of  the  contract  completion  costs. 
Once  compliance  with  the  other  criteria  has  been  established,  the  data  needed  to  calculate 
this  index  is  readily  available.  Although  contractors  are  not  specifically  required  to 
utilize  the  CPIcum  index,  it  definitely  has  merit.  According  to  Christensen  and  Heise, 
“because  the  cumulative  CPI  tends  to  decline,  the  EAC  computed  by  the  cumulative  CPI 
is  a  reasonable  floor  for  the  final  cost  at  completion”  (1993:  13).  In  a  later  study, 
Christensen  provides  strong  evidence  that  the  “CPI-based  EAC  is  a  reasonable  lower 
bound  to  the  final  cost  of  a  defense  contract”  (Christensen,  1996:  46).  This  study  was 
performed  using  data  from  sixty-four  completed  defense  contracts. 

A  good  example  of  this  benefit  can  be  found  within  the  Defense  Contract  Audit 
Agency  (DCAA).  When  the  government  cancels  a  contract  for  its  own  convenience,  it 
must  ensure  that  the  Estimate  at  Completion  (EAC)  is  accurate.  If  it  is  accurate,  the  EAC 
becomes  the  basis  for  termination  costs.  In  today’s  environment  of  shrinking  defense 
budgets,  the  DCAA  may  be  involved  in  more  and  more  contract  terminations  for 
government  convenience  (Presutti,  1993:  53).  Accurate  estimates  can  undoubtedly  save 
the  government  substantial  amounts  of  money  in  termination  costs. 

According  to  phase  one  of  the  Arthur  D.  Little  study, 

more  than  half  of  the  government  business  managers 
surveyed  gave  favorable  ratings  to  the  reports  for  estimates- 
at-completion,  cost  impacts  of  known  problems,  problem 
traceability,  and  schedule  status.  Further,  more  than  half 
the  contractor  program  managers  rated  the  reports 
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favorably  with  respect  to  EAC  and  cost  impacts  of  known 
problems.  (1983: 1II-5) 

Benefit  Seven.  The  utility  of  the  CPIcum  combined  with  the  SPI  to  statistically  forecast 
the  ‘’most  likely’'  range  of  estimated  costs  at  completion  is  yet  another  benefit  of 
C/SC  SC.  Again,  the  first  three  sections  of  the  criteria  make  it  possible  to  perform  the 
earned  value  analyses  that  are  required  in  the  analysis  section.  Although  contractors  are 
not  required  to  use  this  “combined”  method  to  develop  an  estimated  cost  at  completion, 
they  might  want  to  consider  this  method.  In  fact,  many  government  managers  utilize  this 
method  for  calculating  estimates  of  completion  costs.  According  to  Fleming  and 
Koppelman,  combining  both  indexes  allows  us  to  account  for  the  compounding  effects  of 
performance  that  is  both  behind  schedule  and  over  cost  (1996:  27).  The  ability  to  predict 
most  likely  contract  completion  costs  are  useful  for  budgetary  purposes,  or  in  determining 
if  a  project  should  be  terminated  in  favor  of  less  problematic  programs.  Defense  projects 
are  continuously  battling  for  limited  funds,  and  those  encountering  major  problems  are 
often  canceled  prematurely. 

Benefit  Eight.  Benefits  can  also  be  obtained  through  the  use  of  the  To-Complete 
Performance  Index  (TCPI)  to  monitor  the  remaining  project  tasks  against  a  specific 
management  financial  goal.  Basically,  this  index  is  calculated  by  dividing  the  remaining 
work  by  the  remaining  budget  (or  a  revised  budgetary  goal).  Criterion  24,  under  C/SCSC 
(criterion  22  under  EVMS),  requires  the  contractor  to  identify  the  budgeted  cost  of  work 
scheduled  (BCWS),  the  budgeted  cost  of  work  performed  (BCWP),  and  the  actual  cost  of 
work  performed  (ACWP)  on  a  monthly  basis.  After  fulfilling  this  requirement,  the 


contractor,  or  the  government,  has  the  data  necessary  to  calculate  the  TCPI.  This  index 
reflects  how  efficient  the  contractor’s  remaining  performance  must  be  to  recover  from  a 
negative  position.  Christensen  and  Heise  state  “when  the  TCPI  is  significantly  greater 
than  the  cumulative  CPI  and  the  contract  is  beyond  the  20  percent  completion  point,  it  is 
likely  that  the  contract  will  overrun  the  budgetary  goal  despite  any  optimistic  claims  to 
the  contrary”  (1993:  15).  Of  course,  without  the  other  criteria,  the  validity  of  this  index 
would  be  questionable. 

Benefit  Nine.  Another  benefit  of  C/SCSC  relates  to  the  utility  of  a  periodic  CPI  based 
on  the  actual  performance  it  took  to  achieve  a  particular  planned  production  standard. 
Again,  C/SCSC  and  EVMS  require  the  contractor  to  identify  the  BCWS,  BCWP,  and 
ACWP  on  a  monthly  basis  according  to  C/SCSC  criterion  24  (EVMS  criterion  22).  After 
fulfilling  this  requirement,  the  contractor,  or  the  government,  has  the  data  necessary  to 
measure  the  periods’  performance  against  a  predetermined  standard.  Of  course,  without 
the  other  criteria  (related  to  organization,  planning  and  budgeting  and  accounting),  the 
comparison  would  most  likely  be  faulty. 

Benefit  Ten.  The  last  benefit  listed  by  Fleming  and  Koppelman  is  related  to 
Management  by  Exception  (MBE)  principles  that  are  an  inherent  part  of  a  valid  earned 
value  management  control  system.  A  contractor  that  has  complied  with  the  first  three 
sections  of  criteria  related  to  organization,  planning  and  budgeting,  and  accounting,  is 
now  well  suited  to  determine  the  status  of  their  efforts.  In  fact,  the  analysis  criteria 
require  the  contractor  to  perform  analysis  and  explain  what  is  being  done  to  correct 
problems.  However,  instead  of  requiring  the  contractor  to  explain  every  minute 
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deviation.  “Earned  value  project  management  allows  the  project  manager  and  busy 
corporate  executives  to  utilize  the  [MBE]  principle  to  the  fullest”  (Fleming  and 
Koppelman,  1996:  88).  Management’s  attention  can  be  triggered  by  exceptions  to  the 
authorized  plan  without  them  having  to  continuously  monitor  the  smallest  details  of  the 
program. 

Criteria  in  the  last  section  of  C/SCSC  and  EVMS  related  to  revisions  and  access 
to  data  are  very  important  as  well.  Virtually  all  major  projects  will  encounter  changes  to 
the  original  baseline.  In  order  to  comply  with  the  criteria,  the  contractor  must 
"accommodate  such  changes  in  an  orderly  and  controlled  and  documented  manner, 
consistent  with  its  written  procedures”  (Fleming,  1992:  69).  This  ensures  a  proper  audit 
trail  is  available  between  newly  authorized  work  and  the  original  performance  baseline. 

It  also  helps  ensure  the  validity  of  the  data  provided  by  the  contractor.  Therefore,  these 
criteria  are  related  to  all  ten  of  the  benefits  listed  above. 


Other  Benefits 

There  are  other  benefits  of  the  criteria  that  may  not  appear  to  relate  specifically, 

although  they  may  be  linked  ultimately,  to  the  ten  benefits  laid  out  by  Fleming  and 

Koppelman.  For  instance,  in  order  to  comply  with  the  criteria,  contractors  may  be  forced 

to  do  some  things  better  up  front  that  they  might  not  have  done  otherwise.  In  fact,  results 

from  phase  two  of  the  Arthur  D.  Little  study  indicate 

a  large  majority  of  contractor  program  managers  (74%)  and 
government  business  managers  (77%)  also  believe  a  major  benefit  of 
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C/SCSC  is  more  thorough  contractor  planning  than  otherwise  would 
be  accomplished.  (1984:  III-2) 

Government  review  teams  can  also  provide  benefits  to  contractors.  Review  teams 
can  be  of  benefit  “by  drawing  on  their  previous  experience  to  suggest  good  features  of 
other  systems  to  improve  weak  features  of  a  contractor’s  systems”  (Whittenberg,  1972: 
78). 


Other  Evidence  of  Benefits 

According  to  a  study  performed  by  the  Arthur  D.  Little  Program  Systems 
Management  Company, 

the  most  important  overall  conclusion  of  the  study  is  that 
C/SCSC  is  a  valid  concept  and  approach  to  controlling  contract 
performance.  (1984: 1-3) 

This  study  was  conducted  in  an  effort  to  resolve  two  controversies  surrounding  C/SCSC: 
1 )  do  the  costs  of  the  criteria  and  associated  reporting  requirements  outweigh  their 
benefits?,  and  2)  concerns  about  the  specifics  of  how  these  requirements  have  been 
implemented  (Little,  1983: 1-l).  The  study  was  accomplished  in  two  phases:  the  first 
phase  involved  a  mailed  questionnaire,  and  the  second  involved  in-depth  interviews  with 
DoD  and  industry  respondents. 

A  joint  DOD  and  NSIA  report  identified  opportunities  to  improve  the  cost  and 
schedule  management  process.  The  report  stated  “Conclusions  clearly  indicate  that  both 
DoD  and  industry  place  high  importance  on  the  need  for  cost/schedule  management 
control  systems.  They  also  agree  that  change  is  necessary  to  improve  the  quality  and 
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efficiency  of  cost/schedule  performance  measurement  and  reporting”  (1991 :  1-1). 
Additionally,  the  report  found  that,  “More  than  90  percent  of  all  interviewees  (DoD  and 
industry)  place  extremely  high  importance  on  the  need  for  standard  cost/schedule 
management  systems  criteria  as  defined  by  DoDI  7000.2”  (1991;  1-5). 

A  1994  study  requested  by  the  Deputy  Secretary  of  Defense  and  conducted  by  a 
joint  Coopers  &  Lybrand  and  TASC  project  team  looked  at  the  impact  of  DOD 
regulations  and  oversight.  C/SCSC  was  one  of  the  regulatory  areas  identified  by  the 
study  as  having  a  low  added  value.  However,  the  report  states,  “in  general,  industry 
views  the  general  framework  and  principles  of  cost/schedule  reporting  positively” 
(1994:22). 

One  part  of  the  NSIA  survey  asked  which  C/SCSC  concepts  have  been  applied  to 
other  contracts  or  commercial  applications.  Although  responses  indicated  very  few 
contractors  utilized  the  criteria  fiilly  in  these  situations,  various  concepts  were  used  quite 
heavily.  For  instance,  budget  cost  of  work  performed  (BCWP)  was  utilized  by  over  70% 
and  estimate  at  completion  (EAC)  and  reporting  methods  by  over  80%  of  respondents 
(NSIA,  1980:IX-16). 

A  1997  GAO  report  also  adds  credence  to  the  concept  of  earned  value  (GAO, 
1997;  1 1).  The  report  highlights  the  fact  that  commercial  firms  are  adopting  the  concept 
of  earned  value  more  and  more.  Although  many  of  these  new  implementations  of  earned 
value  are  being  applied  in  a  more  streamlined  manner  than  in  the  past,  the  basic  concept 
of  earned  value  is  still  very  much  alive  and  well.  For  example.  United  Defense  Limited 
Partnership  (a  major  defense  contractor)  has  incorporated  earned  value  into  its  entire 
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management  structure.  Also,  Lockheed-Martin  Missiles  and  Space  has  einnounced  it  is 
adopting  earned  value  for  all  its  contracts.  These  include  contracts  with  military  or 
commercial  customers. 

Biery  highlights  the  fact  that  the  military’s  cost  growth  may  not  look  so  bad  when 
compared  with  other  projects  (1985:  12).  In  this  particular  article,  he  attempts  to  quantify 
the  costs  and  benefits  of  competition  in  acquisition  strategies.  It’s  no  secret  that  the 
military  has  experienced  cost  growth  on  many  of  its  contracts.  In  fact,  the  Pentagon  has 
attracted  much  criticism  as  a  result  of  program  cost  growth.  However,  Biery  provides  a 
compelling  table  which  compares  cost  growth  of  major  weapon  systems  with  various 
types  of  complex,  technically  ambitious  non-military  projects.  Figure  3  is  a  graphical 
replication  of  Biery’s  table. 

The  cost  growth  in  Figure  3  is  defined  by  Biery  as  “the  difference  between  the 
actual  costs  (or  the  most  current  estimate  of  actual  costs)  and  the  estimate  made  at  the 
start  of  the  system’s  developmenf’  (1985:  11).  These  growth  factors  were  calculated  in 
constant  dollars  to  eliminate  the  effects  of  inflation.  For  example,  the  military  weapon 
systems  cost  factor  of  1 .4  represents  a  40  percent  cost  overrun.  Further,  Biery  states  that 
the  actual  and  estimated  costs  are  compared  with  one  another  on  a  similar  quantity  basis. 
Biery  also  states  that  “if  one  assumes  that  the  technological  advance  sought  in  many  new 
weapon  systems  is  greater  than  that  generally  sought  in  commercial  and  non-defense 
projects,  then  [the  data  represented  in  this  chart]  may  even  understate  the  military  cost 
record’’ (1985:  12). 
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*Note:  Development  costs  only.  **Note;  Actual  inflation  was  less  than  planned. 

Figure  3.  Cost  Growth  on  Various  Projects  (Biery,  1985: 12) 
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Indeed,  many  public  sector  projects  and  private  sector  projects  exhibited  greater 
(sometimes  significantly  greater)  cost  growth  than  the  military.  Biery  points  to  a  number 
of  management  initiatives  put  into  place  in  the  early  1970s  that  may  have  had  some 
positive  impacts  upon  improving  cost  and  schedule  controls  within  the  military 
acquisition  environment.  Several  policies  under  the  Packard  initiatives  are  mentioned  as 
well  as  the  creation  of  the  Cost  Analysis  Improvement  Group.  However,  Biery  does  not 
mention  specifically  the  possibility  that  earned  value  techniques  within  the  Cost/Schedule 
Control  Systems  Criteria  (C/SCSC)  could  have  been  responsible.  We  believe  that 
C/SCSC  requirements  (that  were  not  present  on  many  of  the  other  contracts)  played  a 
major  role  in  minimizing  cost  growth  on  military  acquisitions. 

The  fact  that  other  U.S.  government  agencies  have  adopted  earned  value  practices 
also  lends  support  for  its  validity  and  merit.  For  example,  the  United  States  Office  of 
Management  and  Budget  requires  “that  cost,  schedule  and  performance  goals  are 
identified  that  can  be  measured  using  an  earned  value  management  system  or  similar 
system”  (0MB,  1997:  3).  The  United  States  Department  of  Energy  adopted  the  C/SCSC 
criteria  in  1975  and  has  imposed  them  on  contractors  doing  business  with  them  ever  since 
(Fleming,  1992:  25).  The  National  Aeronautic  and  Space  Administration  (NASA)  also 
requires  contractors  to  comply  with  the  EVMS  criteria  on  major  acquisitions  as  described 
in  Policy  Directive  NPD  9501  3. 

The  governments  of  Australia,  Canada  and  Sweden  are  incorporating  earned  value 
techniques  into  many  of  their  projects. 
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At  the  Trilateral  Defense  Industrial  Cooperation 
Forum  held  in  Washington,  D.C.  in  September 
1992,  Australia,  Canada  and  the  United  States 
agreed  to  set  up  the  International  Performance 
Management  Council  (IPMC).  IPMC  objectives  are 
1 )  to  exchange  information  at  the  policy  and 
administrative  levels  on  respective  C/SCSC 
programs,  2)  to  strive  toward  reciprocal  recognition 
of  contractors  accepted  as  C/SCSC-compliant  by 
each  country,  and  to  promote  internationally  the 
“earned  value”  management  techniques  embodied 
in  C/SCSC...  Since  then,  Sweden,  New  Zealand, 
and  the  United  Kingdom  have  agreed  to  become 
members  of  the  IPMC.  (IPMC,  1997) 


For  more  evidence  of  the  merit  of  earned  value,  we  could  certainly  examine  the 
utilization  rates  of  the  reports  generated  by  a  C/SCSC  compliant  system.  In  the  NSIA 
surv  ey,  participants  were  asked  what  levels  of  senior  management  use  their  cost 
performance  reports  (with  their  earned  value  measurements)  for  status  information.  The 
respondents  indicated  the  report  is  utilized  by:  99%  of  contract  program  managers;  57% 
of  functional  branch  managers;  55%  of  finance  controllers;  50%  of  company  vice- 
presidents,  as  well  as  small  portions  of  corporate  vice  presidents  and  other  managers 
(NSIA,  1980: 1-25). 

One  particular  example  of  an  extremely  successful  organization  using  an  earned 
value-based  management  information  system  is  at  Levi  Strauss’  Albuquerque  plant 
(Hatfield,  1996:  27).  Each  machine  used  to  process  material  keeps  track  of  actual  costs 
and  time  as  well  as  the  number  of  garments  processed  and  the  extent  to  which  they  have 
been  finished.  This  system  generates  a  report  resembling  a  Cost  Performance  Report 
(CPR)  every  hour.  Through  the  use  of  tight  variance  thresholds,  the  company  is  able  to 
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utilize  management-by-exception  to  a  very  high  degree.  Positive  variances  are  used  as  a 
basis  for  employee  bonuses.  Earned  value  management  was  such  an  integral  part  of  their 
system,  managers  and  floor  supervisors  say  they  could  not  imagine  running  their  business 
without  these  hourly  status  reports. 

Other  evidence  of  earned  value  benefits  can  be  found  in  a  letter  published  in 
Program  Managers  Newsletter  (Feedback,  1978).  This  letter  discusses  the  cost 
avoidance  due  to  improved  discipline  and  management  visibility  from  the 
implementation  of  C/SCSC.  The  government  had  decided  to  delete  a  scheduled  task 
from  the  contract  and  requested  a  credit  for  the  reduction  in  scope.  Since  none  of  the 
work  being  deleted  had  been  started,  the  only  obstacle  was  determining  the  value  of  the 
work.  The  work  had  been  planned  in  the  contractors  C/SCSC  system,  so  preparation  of 
the  proposal  was  almost  effortless.  Additionally,  government  review  of  the  proposal  was 
also  simplified.  The  contractor  and  government  were  less  than  two  percent  apart  at  the 
start  of  negotiations,  and  settled  on  a  price  in  minutes.  The  savings  were  quantified  by 
assessing  the  proposal  preparation,  fact  finding,  and  negotiation  times.  Total  cost  of 
proposal  preparation  was  only  $200,  compared  to  past  proposal  costs  of  $10,000.  The 
government  used  only  eight  person  hours  for  fact  finding  versus  the  usual  100. 
Negotiations  usually  take  100  person  hours,  so  total  government  savings  (fact  finding  and 
negotiations)  is  estimated  at  $5,000.  The  total  savings  was  estimated  to  be  approximately 
$15,000  (Feedback,  1978:22). 

In  another  case,  a  program  with  two  competing  developers  was  canceled.  The 
associated  problem  was  in  determining  the  target  cost  which  the  contractors’  fees  would 
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be  based  on.  Using  reports  and  planning  from  the  contractor’s  C/SCSC  validated  system, 
the  project  office  and  one  contractor  reached  agreement  instantaneously,  while  it  took 
three  days  for  the  other  contractor.  The  immediate  savings  to  the  government  was 
$400,000,  while  the  manpower  savings  may  have  been  greater  since  a  claims  situation 
was  avoided  (Feedback,  1978:  22). 

Future  Benefits 

There  are  many  other  benefits  that  could  be  realized  in  the  future  from  the  use  of 
criteria-compliant  systems.  For  instance,  validated  earned  value  data  could  be  used  to 
determine  progress  payments  instead  of  the  traditional  percent  complete  approach.  The 
percent  complete  approach  has  historically  been  used  (without  regard  to  actual  work 
performed  to  date).  No  doubt,  this  method  could  very  easily  distort  work  accomplished 
and  result  in  progress  overpayments  (or  underpayments). 

Fleming  and  Koppelman  agree  that  “the  fundamental  guiding  principle  that 
should  apply  to  all  payments  being  made  from  the  owner  of  a  project  to  the  contractors 
performing  the  project  work  [is  that]  all  such  payments  should  be  made  in  concert  with 
the  physical  work  accomplished  on  the  project”  (Fleming  and  Koppelman  1995:  C.2.1). 
Further,  they  suggest  that  all  payments  to  the  contractor  be  made  at  some  value  less  than 
1 00%  of  the  total  costs  incurred  which  provides  the  contractor  incentives  to  complete  the 
project  (Fleming  and  Koppelman,  1995,  C.2.4). 
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Areas  outside  of  the  defense  environment  could  also  benefit  from  the  use  of 


earned  value.  For  instance,  Fleming  and  Koppelman  offer  a  simplified  approach  of 
incorporating  earned  value  into  various  types  of  projects  (1996:  1 12).  Although  the  value 
of  this  approach  has  yet  to  be  fully  tested,  it  would  no  doubt  be  faster,  easier,  and  less 
costly  to  implement  than  the  full  blown  version. 

Nancy  Singletary  also  suggests  other  uses  of  earned  value  techniques.  She  states 
“no  longer  just  a  method  for  meeting  C/SCSC  standards  for  the  Department  of  Defense, 
Earned  Value  methodology  is  holding  its  own  and  paying  its  way  in  commercial 
industries,  public  and  private  sectors,  manufacturing,  financial  institutions, 
pharmaceuticals,  and  other  industries  interested  in  optimizing  the  profit  line  or  just 
breaking  even”  (1996:  28-29).  Singletary  also  agrees  that  a  major  benefit  of  earned  value 
is  the  ability  to  compare  the  progress  of  dissimilar  projects  (1996:  28-29). 

Presutti  echoes  the  idea  that  earned  value  could  enhance  the  management 
capabilities  of  organizations  outside  the  Defense  industry.  Plus,  he  adds,  the 
documentation  for  this  government  process  is  readily  available  and  virtually  free  (1993: 
53). 


Summary 

Chapter  II  described  the  costs  associated  with  complying  with  C/SCSC.  In  this 
chapter  we  have  attempted  to  show  benefits  that  may  result  from  a  contractor’s  use  of 
C/SCSC  on  acquisition  programs.  Now  that  we  have  looked  at  the  costs  and  benefits  of 
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the  criteria  separately,  in  the  next  chapter  we  will  look  at  them  together  in  order  to  better 
understand  their  relationship. 


52 


IV.  Cost  vs.  Benefits  and  the  Implementation  of  EVMS 


C/SC  SC  Costs  versus  Benefits 


In  Chapters  II  and  III,  we  identified  the  costs  and  benefits  of  C/SCSC.  Before  we 
compare  the  costs  and  benefits,  we  will  review  conclusions  from  previous  studies  on  this 
subject.  Then,  we  will  summarize  our  findings  on  the  costs  and  benefits  of  the  criteria. 
Next,  we  will  show  how  data  from  criteria-compliant  management  control  systems  can  be 
viewed  within  the  framework  of  the  usefulness  of  the  information  it  provides.  We  will 
describe  some  of  the  veirious  qualities  of  information;  and  how  tradeoffs  must  be  made  by 
managers  in  determining  how  to  maximize  the  usefulness  of  information  for  all 
customers.  Finally,  we  will  determine  how  the  costs  and  benefits  are  affected  by  the 
switch  to  EVMS. 

Several  studies,  including  the  Arthur  D.  Little  study  and  the  Coopers  & 

Lybrand/TASC  study,  have  made  conclusions  about  the  costs  and  benefits  of  C/SCSC.  A 

National  Security  Industrial  Association  (NSIA)  report  cited  that. 

Although  C/SCSC  performance  measurement  criteria  have  been  in  existence  for 
over  1 5  years,  contractors  still  do  not  find  them  sufficiently  productive  or  cost 
effective  to  use  to  any  substantial  extent  in  managing  their  business  unless  forced 
to  do  so  by  contractual  requirement.  What  little  is  used  voluntarily  for  analysis 
and  control  is  applied  at  significantly  higher  levels  in  the  cost  isolation  structure 
than  is  usually  demanded  under  C/SCSC  requirements  imposed  by  customer  or 
SAR  team  interpretation  of  the,  criteria.  Apparently  this  limited  application  is 
judged  to  be  considerably  more  cost  effective,  less  burdensome  documentation- 
wise,  yet  effective  in  management  of  the  business  (1980:  16).  Another  important 
finding  concludes  that,  “While  many  contractors  have  voluntarily  adopted  the 
basic  concept  of  earned  value  performance  measurement,  no  more  than  4% 
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indicate  that  they  voluntarily  use  the  full  system  that  they  are  required  to  subject 
to  Government  procedural  interpretations  and  acceptance  conditions.  (1980:  18) 

Phase  I  of  the  Arthur  D.  Little  study  concluded  that,  . .  the  majority  of 
respondents  within  each  of  the  four  populations  believe  that  C/SCSC  benefits  to 
themselves  outweigh  its  costs.  However,  the  majorities  were  not  large.  They  ranged 
from  53%  to  62%  in  the  four  populations”  (1983: 1-2). 

Additionally,  this  study  questioned  the  usefulness  of  cost  reports  generated  from 
C/SCSC-approved  systems  and  found  that,  “  . . .  close  to  four-out-of-five  respondents 
rated  the  reports  as  being  either  good  or  excellent  in  helping  to  determine  the  cost  status 
of  their  contracts.  However,  for  helping  to  determine  aspects  of  contract  status  other  than 
cost,  less  than  half  of  the  government  program  managers  rated  the  reports  as  either  good 
or  excellent”  (1983:  1-2). 

Phase  II  of  the  Arthur  D.  Little  study  added  to  the  surveyed  information  from 

Phase  I  by  conducting  in-depth  interviews.  Phase  II  concluded  that,  “The  most  important 

overall  conclusion  of  the  study  is  that  C/SCSC  is  a  valid  concept  and  approach  to 

controlling  contractor  performance”  (1984: 1-3).  Phase  II  also  noted  that. 

All  four  categories  of  managers  agreed  (ranging  from  53%  to  62%)  that  C/SCSC 
benefits  to  themselves  outweighed  its  associated  cost.  However,  the  strength  of 
this  belief  was  less  than  that  relating  solely  to  C/SCSC  effectiveness.  The  pattern 
of  the  data . . .  appeared  to  suggest  that  perceptions  of  whether  benefits  outweigh 
costs  were  relatively  elastic.  In  addition,  these  data  appeared  to  be  inversely 
related  to  the  cost  of  system  operation  . . .  (1984:  III-2) 

A  joint  DOD/NSIA  report  identified  opportunities  to  improve  the  cost  and 
schedule  management  process.  The  report  found  that,  “Conclusions  clearly  indicate  that 
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both  DoD  and  industry  place  high  importance  on  the  need  for  cost/schedule  management 
control  systems.  They  also  agree  that  change  is  necessary  to  improve  the  quality  and 
efficiency  of  cost/schedule  performance  measurement  and  reporting”  (1991:  1-1). 
Additionally,  the  report  found  that,  “More  than  90  percent  of  all  interviewees  (DoD  and 
industry)  place  extremely  high  importance  on  the  need  for  standard  cost/schedule 
management  systems  criteria  as  defined  by  DoDI  7000.2”  (1991:  1-5). 

A  1994  study  requested  by  the  Deputy  Secretary  of  Defense  and  conducted  by  a 
joint  Coopers  &  Lybrand  and  TASC  project  team  looked  at  the  impact  of  DOD 
regulations  and  oversight.  C/SCSC  ranked  third  among  the  top  ten  cost  drivers,  however 
only  accounted  for  0.9%  of  the  total  ‘value  added’  costs  (total  contract  costs  less  material 
costs).  The  study  did  not  look  at  the  benefits  associated  with  C/SCSC  regulatory 
compliance,  but  does  state. 

In  general,  industry  views  the  general  framework  and  principles  of  cost/schedule 
reporting  positively.  Sound  program  management  requires  regular  analysis  of 
expenditure  and  performance  trends,  and  deviations  must  be  addressed  early  to 
avoid  overruns  and  delays.  However,  all  contractors  subject  to  C/SCSC  agree 
that,  as  currently  required  by  DoD,  cost/schedule  reporting  is  too  detailed, 
repetitive,  and  voluminous  to  be  used  effectively  as  a  management  tool  by  either 
the  government  or  industry,  and  that  the  requirement  may  in  fact  undermine 
program  performance  by  diverting  the  time  and  attention  of  the  company  program 
manager.  (1994:  22) 

A  1997  GAO  report  presented  a  brief  history  of  C/SCSC,  identified  problems  with 

C/SCSC,  and  discussed  means  to  overcome  these  problems,  including  the  switch  to  the 

industry  EVMS.  The  report  stated. 

The  core  concept  of  the  CS\2  process-earned  value— is  recognized  as 
a  sound  way  to  measure  progress  on  major  acquisition  programs.  Over 
the  years,  however,  the  process  has  evolved  to  where  the  needs  of 
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some  of  its  key  users  are  being  satisfied,  while  others  are  not. 
Specifically,  DOD  program  managers  are  not  satisfied  with  the 
timeliness  of  the  CS\2  reports.  Because  the  data  contained  in  the 
reports  are  typically  up  to  2  months  old,  the  reports  do  not  fiinction 
as  an  early  warning  system  needed  by  program  managers.  Moreover,  the 
process  has  not  fully  integrated  cost,  schedule,  and  technical  data 
as  intended.  The  want  of  such  information  can  invite  subjective  and 
potentially  optimistic  judgments  to  fill  the  void.  Contractors 
maintain  that  accommodating  extensive  government  certification 
reviews,  collecting  and  arraying  data  in  prescribed  categories,  and 
preparing  detailed  reports  requires  significant  effort  and  cost  to 
the  government  and  draws  some  of  their  engineering  resources  away 
from  program  execution.  Commercial  firms  that  use  earned  value 
systems  produce  reports  more  frequently,  more  quickly,  and  in  less 
detail  than  the  CS\2  process.  Users  outside  the  program 
offices— such  as  financial  managers  and  cost  estimators— find  that 
the  data  generally  meets  their  needs.  These  users  generally  place 
more  value  on  consistency  among  cost  categories  and  less  value  on 
timeliness  than  program  managers  (3). 


In  Chapters  II  and  III,  we  concluded,  separately,  the  costs  and  benefits  of 
C/SCSC.  The  summarized  costs  and  benefits  may  be  looked  at  together  in  Tables  8  and 
9. 


Tables.  C/SCSC  Costs 


C/SCS-Compliant 

Costs 

‘Normal’  System 
Compliance 
Costs 

Marginal  Cost  of 
C/SCSC 
Compliance 

Person  Days 

Person  Days 

Person  Days 

System  Compliance  Costs 

1,086 

605-752 

334-481 

System  Design  Costs 

252 

252 

0 

System  Documentation  Costs 

101 

50 

50 

Training  Costs 

184 

184 

0 

System  Implementation  Costs 

367 

110-257 

110-257 

System  Evaluation  Costs 

184 

9 

174 

System  Operating  Costs  (These 
costs  are  in  hours) 

[40  +  (2  *  #  of  CAs)] 

*  (#  of  Post  Validation 
CPR  cycles) 

50%  of  the 
C/SCSC- 
Compliant  Costs 

50%  of  the 
C/SCSC- 
Compliant  Costs 
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Table  9.  Benefits  of  C/SCSC 


1. 

Employment  of  a  single  management  control  system. 

2. 

Integration  of  cost,  schedule  and  technical  parameters. 

3. 

Resulting  database  that  reflects  a  consistent  and  predictable  performance  history. 

4. 

Utility  and  stability  of  the  cumulative  Cost  Performance  Index  (CPI  cum)  to 
monitor  performance. 

5. 

The  utility  of  the  Schedule  Performance  Index  (SPI)  to  monitor  and  quantify  the 
value  of  the  work  scheduled  against  work  performed. 

6. 

Utility  of  the  CPIcum  to  statistically  forecast  low  end  range  of  completion  costs. 

7. 

Utility  of  the  CPIqum  combined  with  SPI  to  forecast  most  likely  completion  cost. 

8. 

Utility  of  the  To-Complete  Performance  Index  (TCPI)  to  monitor  remaining 
tasks. 

9. 

Utility  of  a  periodic  CPI  to  measure  performance  against  a  pre-set  standard. 

10. 

The  use  of  Management  by  Exception  principles  to  focus  management’s  attention 
on  significant  exceptions  to  the  authorized  plan. 

11. 

The  validity  and  accuracy  of  the  above  mentioned  “earned  value  benefits”  are 
empowered  by  requirements  embedded  in  the  criteria. 

12. 

The  ability  to  use  valid  earned  value  data  to  determine  progress  payments. 

13. 

A  management  discipline  that  may  require  contractors  to  do  things  better  up  front 
than  they  might  have  otherwise  done. 

14. 

Evidence  of  smaller  cost  growth  in  the  military  than  in  many  other  commercial 
and  non-DoD  projects. 

Summary  of  C/SCSC  Costs  versus  Benefits 

So  how  do  the  costs  and  benefits  of  the  criteria  relate?  We  encountered  much 
difficulty  at  first  in  our  attempts  to  analyze  both  together  in  order  to  arrive  at  some 
conclusion  about  the  overall  utility  of  the  criteria.  One  approach  compares  the  cost  of 
C/SCSC  to  the  cost  overruns  of  projects  not  complying  with  C/SCSC. 

Table  3,  C/SCSC  System  Compliance  Costs  (%  of  Total  Contract  Value), 
identified  several  studies  that  estimated  the  costs  of  C/SCSC  as  a  percentage  of  total 
contract  value.  The  range  of  the  cost  of  C/SCSC  was  from  0.1%  -  5%  of  total  contract 
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value.  Figure  3,  Cost  Growth  on  Various  Projects,  compares  the  cost  growth  of  non-DoD 
projects  to  DoD  project  cost  growth.  The  associated  1985  paper  by  Biery  found  that  DoD 
projects  on  average  experience  a  40%  cost  overrun.  The  average  overrun  of  the  non-DoD 
projects  identified  in  the  paper  was  128%,  with  a  range  of  8%  to  593%  and  a  median  of 
70%.  While  it  can  not  be  determined  if  the  difference  is  due  to  C/SCSC  compliance,  or 
other  factors,  it  does  indicate  that  0.1%  to  5%  of  total  contract  value  is  a  small  price  to 
pay  for  the  associated  C/SCSC  benefits. 

Another  approach  that  adds  structure  to  this  analysis  goes  back  to  basic 
accounting  concepts.  One  of  the  main  purposes  of  the  criteria  is  to  ensure  that  contractor 
management  control  systems  provide  data  that  can  be  transformed  into  useful  information 
for  decision  makers. 

The  characteristics  of  information  that  make  it  a  desirable 
commodity  can  be  viewed  as  a  hierarchy  of  qualities,  with 
usefulness  for  decision  making  of  most 
importance. . .Relevance  and  reliability  are  the  two  primary 
qualities  that  make  accounting  information  useful  for 
decision  making. .  .To  be  relevant,  information  must  be 
timely  and  it  must  have  predictive  value  or  feedback  value 
or  both.  To  be  reliable,  information  must  have 
representational  faithfulness  and  it  must  be  verifiable  and 
neutral.  Comparability,  which  includes  consistency,  is  a 
secondary  quality  that  interacts  with  relevance  and 
reliability  to  contribute  to  the  usefulness  of  information. 

Two  constraints  are  included  in  the  hierarchy,  both 
primarily  quantitative  in  character.  Information  can  be 
useful  and  yet  be  too  costly  to  justify  providing  it.  To  be 
useful  and  worth  providing,  the  benefits  of  information 
should  exceed  its  cost.  (FASB,  1993:  27) 

Christensen  developed  a  chart  based  upon  this  concept  from  the  FASB  (1989:  30). 
His  adaptation  is  tailored  to  the  defense  environment  and  is  replicated  in  Figure  4. 


58 


Figure  4.  Information  Quality  and  the  Criteria 


Let  us  view  government  regulatory  requirements,  such  as  C/SCSC  or  EVMS 
within  the  framework  of  Figure  4.  One  of  the  main  products  of  a  criteria-compliant 
system  is  cost,  schedule,  and  technical  information.  At  the  top  of  the  information  quality 
hierarchy  is  decision  usefulness.  If  the  information  produced  is  not  useful  for  decision 
makers,  then,  of  course,  it  adds  costs  but  no  benefits  to  the  process.  One  of  the 
difficulties  associated  with  C/SCSC  or  EVMS  is  that  it  must  serve  several  decision 
makers.  Contractors,  program  managers,  financial  managers,  engineers,  program  element 
managers  (PEM),  program  executive  officers  (PEO),  DCMC  personnel,  and 
Congressional  members  are  all  decision  makers  that  use  the  information  generated  from  a 
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criteria-compliant  management  control  system.  The  amount  and  type  of  information 

needed  by  these  different  decision  makers  varies  greatly.  A  financial  manager  may  be 

interested  in  the  costs  of  WBS  level  three  elements,  while  a  PEO  may  only  be  concerned 

with  costs  at  the  total  program  level.  A  GAO  report  details  this  concept: 

An  earned  value  system  faces  stiff  and  somewhat  competing  demands  from  its 
users:  (1)  providing  the  right  analyses  in  time  for  program  managers  to  use;  (2) 
enabling  adequate  oversight  and  analysis  of  multiple  programs  beyond  the 
program  office  level;  and  (3)  minimizing  the  effort  required  of  the  contractor  to 
provide  the  necessary  systems,  data,  and  analysis.  (1997:  4) 

The  majority  of  these  different  needs  must  be  met  while  balancing  two  very 

important  characteristics:  relevance  and  reliability. 

Information  relevance  and  reliability  is  a  continuum  where  one  is  traded  for  the 

other.  In  order  to  obtain  more  reliable  information,  we  may  be  forced  to  sacrifice  some  of 

the  relevance  of  that  information  and  vice  versa.  For  example,  if  we  want  monthly  cost 

information  to  be  audited  by  an  independent  source,  we  gain  reliability,  but  lose 

relevance  as  the  auditing  time  will  delay  the  delivery  of  the  information  to  the  decision 

maker.  This  also  ties  into  the  variety  of  decision  makers’  needs.  A  program  manager’s 

information  needs  may  stress  timely  cost  information,  while  a  financial  manager’s  may 

stress  valid  cost  information. 

If  we  consider  the  relevance  portion  of  the  diagram  to  be  related  to  the  benefits  of 
the  criteria  and  the  reliability  portion  to  the  costs,  then  we  can  make  some  useful  points. 
The  implementation  of  the  criteria  generate  costs  that  should  be  justified  by  the  reliability 
of  the  information  produced  by  a  compliant  system.  From  these  compliant  systems, 
reliable  and  relevant  information  is  passed  on  to  the  user.  As  the  quest  for  ‘reliable’ 
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information  proceeds,  costs  increase,  but,  equally  important,  the  relevance  of  the 
information  starts  to  weaken.  More  reliable  information  can  translate  into  more 
complicated  management  control  systems,  more  detailed  reporting,  more  information 
auditing,  and  higher  costs.  The  reliability  of  the  information  is  increased  at  the  sacrifice 
of  the  relevance.  Since  cost  reporting  takes  longer,  and  more  detailed  information  is  in 
the  cost  reports,  it  becomes  harder  for  the  user  to  filter  out  the  needed  information. 
Eventually,  this  leads  to  a  very  reliable  management  control  system  that  is  not  as  useful 
to  the  decision  makers.  On  the  other  hand,  a  non-criteria-compliant  management  control 
system  can  generate  top  level  cost  information  hourly,  without  being  audited,  but  is  the 
information  reliable?  It  is  easy  to  see  how  the  delicate  balance  between  reliability  and 
relevance  can  affect  the  costs  and  benefits  of  C/SCSC  or  EVMS,  especially  coupled  with 
the  differences  in  user’s  needs. 

Based  on  our  research  and  the  GAO’s  findings,  we  believe  that  the  government 
progressively  moved  toward  reliability  and  has  lost  some  relevance  in  the  process.  This 
may  have  resulted  in  additional  costs,  and  degraded  the  relevance  of  the  information 
provided  to  managers.  However,  in  light  of  recent  reform  initiatives,  the  focus  seems  to 
be  shifting  back  towards  relevance.  We  are  not  suggesting  that,  overall,  the  costs  have 
outweighed  the  benefits.  We  simply  believe  that  in  its  attempt  to  monitor  and  manage 
contracts,  the  government  has  focused  more  on  reliability  than  was  absolutely  necessary. 
A  1997  GAO  report  says,  “  . . .  CS^  has  been  viewed  by  other  users  as  a  compulsory  and 
burdensome  financial  reporting  system.  Moreover,  it  has  not  fully  satisfied  the  need  of 


61 


many  program  managers  for  up-to-date  and  integrated  information  on  cost,  schedule,  and 
technical  progress”  (1997:4). 

It  is  no  secret  that  many  major  DoD  acquisition  programs  have  experienced  cost 
overruns.  However,  as  we  pointed  out  earlier,  the  DoD’s  cost  growth  has  been  much 
lower  than  other  major  non-defense  projects.  But  any  process  that  utilizes  public  funds 
inefficiently  should  be  scrutinized  in  hopes  of  eliminating  wasteful,  inefficient  practices. 
The  enormous  monetary  values  of  many  defense  contracts  emphasize  this  point.  For 
instance,  a  ten  percent  overrun  on  a  contract  may  not  sound  so  terrible  to  some 
individuals.  Contrarily,  if  the  ten  percent  overrun  is  on  a  one  hundred  million  dollar 
contract,  the  overrun  of  ten  million  dollars  now  sounds  much  more  significant.  Small 
overruns  are  also  not  acceptable  due  to  the  sheer  number  of  defense  contracts. 
Collectively,  these  overruns  can  have  a  dramatic  effect  upon  our  resources  and  ultimately 
upon  our  defense  capability. 

As  a  result  of  past  acquisition  problems,  the  DoD  has  implemented  more  and 
more  oversight  mechanisms.  One  could  argue  that  the  requirements  of  C/SCSC  could  not 
have  become  more  burdensome  since  the  35  criteria  have  remained  virtually  unchanged 
since  their  inception  in  1967.  However,  the  way  in  which  these  criteria  have  been 
interpreted  and  enforced  may  have  become  more  burdensome  to  the  contractor  than  was 
necessary. 
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EVMS  -  What  Has  Changed  From  C/SCSC? 


In  August  1996,  five  industry  associations  published  the  Industry  Standard 
Guidelines  for  Earned  Value  Management  Systems  (EVMS),  as  a  replacement  to  the 
C/SCSC  criteria.  The  DoD  formally  recognized  the  guidelines  of  EVMS  in  December 
1996  (GAO,  1997:  16). 

The  32  EVMS  criteria  are  very  similar  to  the  35  C/SCSC  criteria.  For  example, 
both  sets  of  criteria  are  divided  into  five  broad  categories:  organization;  planning, 
scheduling  and  budgeting;  accounting  requirements;  analysis  and  management  reports; 
and  revisions  and  data  maintenance.  However,  there  are  some  differences  between  the 
two  sets  of  criteria  worth  noting.  Comparison  of  the  specific  criteria  is  provided  in 
Appendix  A. 

Essentially,  the  primary  difference  between  the  two  sets  of  criteria  lies  in  the 
acceptance  procedures.  Under  C/SCSC,  the  government  had  to  formally  validate  and 
accepted  a  contractor’s  management  control  system.  The  Industry  Standard  Guidelines 
for  Earned  Value  Management  Systems  explains  the  self-certification  concept.  The 
guidelines  state, 

The  process  of  self  certification  involves  written  corporate  assurance  that  the 
company  program  management  system  meets  the  full  intentions  of  the  guidelines 
presented  in  this  EVMS  industry  standard.  The  company  is  responsible  for 
evaluation  of  its  system  and  the  certificate  of  self  certification  should  be  signed  by 
the  chief  executive  officer  of  the  company  involved.  (Section  5) 

The  company  can  use  outside  resources  such  as  peer  groups,  consultants,  and 
government  to  assist  in  the  self  certification. 


63 


A  1997  GAO  report  cites,  “DOD  does  not  accept  the  self-certification  provisions 
of  the  EVMS  standard.  Instead,  it  would  like  to  find  some  middle  ground  between  self- 
certification  and  traditional  government  certification”  (17).  The  report  also  notes  that 
specific  data  access  rights,  and  the  contractor’s  ability  to  change  the  baseline  with  only 
notification  rather  than  approval  are  two  other  differences  between  C/SCSC  and  the  new 
EVMS  criteria  (17). 

The  Earned  Value  Implementation  Guide  provides  guidance  on  how  the  self- 
certification  issue  is  to  be  handled.  The  guide  states,  “For  all  contracts  requiring 
compliance  with  the  criteria,  the  validity  of  the  PMB  will  be  sustained  through  the 
conduct  of  an  Integrated  Baseline  Review  (IBR)”  (1996:  36).  The  guide  goes  into  further 
detail  about  the  objectives  and  approach  of  the  IBR,  and  also  discusses  the  compliance 
evaluation  policy. 

When  the  application  of  the  Criteria  is  required,  it  is  policy  to  ensure  that:  (1)  no 
changes  to  contractors'  existing  EVMSs  are  required  except  those  necessary  to 
meet  the  intent  of  the  criteria;  (2)  the  contractor  has  properly  implemented  the 
EVMS  on  the  contract  under  review  and  is  using  it  as  a  mainstream  program 
management  tool;  and,  (3)  the  contractor  is  using  the  data  from  its  own  EVMS  in 
reports  to  the  government.  (1996:  38) 


Figure  5  shows  that  the  choice  of  self-certification  rests  on  the  contractor,  but  the 
government  must  still  accept  the  self-certification  .  The  guide  defines  the  conduct  of  the 
compliance  evaluation,  which  is  required  if  the  contractor  chooses  to  allow  the 
government  to  review  and  validate  its  management  control  system. 
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Figure  5.  EVMS  Post  Contract  Process 
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Further  guidance  on  the  EVMS  criteria  are  found  in  changes  to  the  FAR  by  the 
Defense  Federal  Acquisition  Regulation  Supplement.  One  specific  change  addresses  the 
self-certification  issue: 

7.  Section  252.234-7001  is  revised  to  read  as  follows:  252.234-7001  Earned 
value  management  system.  As  prescribed  in  234.005-71 ,  use  the  following 
clause:  Earned  Value  Management  System  (Mar  1997) 

7(a)  In  the  performance  of  this  contract,  the  Contractor  shall  use  an  earned 
value  management  system  (EVMS)  meeting  the  criteria  provided  in  DoD  5000.2- 
R,  Mandatory  procedures  for  Major  Defense  Acquisition  Programs  and  Major 
Automated  Information  Systems. 

7(b)  If  the  Contractor  has  an  EVMS  that  has  been  recognized  by  the  cognizant 
Administrative  Contracting  Officer  (ACO)  as  eomplying  with  the  EVMS  criteria 
(or  an  existing  cost/schedule  control  system  (C/SCS)  that  has  been  accepted  by 
the  Government),  the  Contractor  shall  apply  the  system  to  this  contract  within  60 
calendar  days  after  contract  award  or  as  otherwise  agreed  to  by  the  parties. 

7(c)  If  the  Contractor  does  not  have  an  EVMS  that  has  been  recognized  by  the 
cognizant  ACO  as  complying  with  EVMS  criteria  (or  does  not  have  an  existing 
C/SCS  that  has  been  accepted  by  the  Government),  the  Contractor  shall  be 
prepared  to  demonstrate  to  the  ACO  that  the  EVMS  complies  with  the  EVMS 
criteria  referenced  in  paragraph  (a)  of  this  clause. 

7(d)  The  Government  may  require  an  integrated  baseline  review  within  180 
calendar  days  after  (1)  contract  award,  (2)  the  exercise  of  significant  contract 
options,  or  (3)  the  incorporation  of  major  modifications.  The  purpose  of  the 
integrated  baseline  review  is  for  the  Government  and  the  Contractor  to  jointly 
assess  areas,  such  as  the  Contractor's  planning,  to  ensure  complete  coverage  of  the 
statement  of  work,  logical  scheduling  of  the  work  activities,  adequate  resourcing, 
and  identification  of  inherent  risks. 

7(e)  Unless  a  waiver  is  granted  by  the  ACO,  Contractor  proposed  EVMS 
changes  require  approval  of  the  ACO  prior  to  implementation.  The  ACO  shall 
advise  the  Contractor  of  the  acceptability  of  such  changes  within  30  calendar  days 
after  receipt  of  the  notice  of  proposed  changes  from  the  Contractor.  If  the  advance 
approval  requirements  are  waived  by  the  ACO,  the  Contractor  shall  disclose 
EVMS  changes  to  the  ACO  at  least  14  calendar  days  prior  to  the  effective  date  of 
implementation. 
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7(f)  The  Contractor  agrees  to  provide  access  to  all  pertinent  records  and  data 
requested  by  the  ACO  or  duly  authorized  representatives.  Access  is  to  permit 
Government  surveillance  to  ensure  that  the  EVMS  complies,  and  continues  to 
comply,  with  the  criteria  referenced  in  paragraph  (a)  of  this  clause. 

7(g)  The  Contractor  shall  require  those  subcontractors  specified  in  the  contract 
for  application  of  the  EVMS  criteria  to  comply  vvdth  the  requirements  of  this 
clause. 

The  previous  examples  describe  how  the  DoD  is  implementing  the  self- 
certification  of  the  EVMS  standards.  The  question  now  becomes:  how  will  the  costs  and 
benefits  be  impacted  by  the  new  EVMS  criteria?  It  is  easy  to  see  that  system  evaluation 
costs  will  decrease  as  a  result  of  the  change  to  EVMS,  but  will  the  benefits  be  affected? 
Benefits  eleven  and  twelve  from  Table  7  are  the  only  benefits  that  may  be  affected  by  the 
switch  to  EVMS,  specifically  the  self-certification  provision.  One  may  argue  that 
without  a  full  government  review  and  validation  of  a  contractor’s  management  system, 
data  from  said  system  will  not  be  valid.  However,  DCMC’s  compliance  policy  (detailed 
in  Figure  3)  for  validating  the  contractor’s  management  system  and  its  corresponding  data 
is  intended  to  ensure  that  the  EVMS  criteria  are  met.  As  seen  in  Figure  5,  step  8  allows 
the  government  to  determine  if  the  contractor’s  self-evaluation  of  their  criteria-compliant 
management  control  system  is  acceptable.  If  the  EVMS  criteria  are  met,  the  data  from 
the  validated  management  control  system  should  be  valid.  Thus,  we  conclude  the  change 
to  EVMS  will  reduce  the  C/SCSC  costs  while  not  affecting  the  associated  benefits. 

Besides  the  switch  to  EVMS,  there  are  many  other  changes  that  have  come  about 
as  a  result  of  acquisition  reform.  The  DoD  is  moving  more  toward  commercial  practices 
in  an  attempt  to  improve  the  acquisition  process  and  save  money.  Some  of  the  major 
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differences  between  commercial  earned  value  practices  and  traditional  defense  practices 
are  outlined  below: 


Table  10.  Commercial  Versus  DoD  Earned  Value  Practices  (GAO,  1997) 


Characteristic 

Commercial  Program 

Traditional  DoD  Program 

Frequency  of  status  reports 

W  eekly /bi-weekly 

Monthly 

Age  of  information 

Real-time/weekly 

Up  to  60  days  after 
reporting  period  ends 

Method  of  data  dissemination 

Direct  access  to  database 

Mailing  of  reports  on  paper 
or  disk 

Work  breakdown  structure 

Level  3 

Level  3-7 

Variance  analysis  reporting 

Critical  path  items 

All  elements 

Management  focus 

Technical  and  schedule 

Costs 

Status  reports  that  are  generated  more  frequently  should  result  in  more  timely 
information  and  allow  management  to  be  more  responsive.  The  ability  to  access  the 
contractor’s  database  directly  could  also  enhance  management’s  responsiveness.  Also,  it 
could  minimize  transcription  and  other  errors  that  could  occur  from  the  previous  method 
of  mailing  paper  reports  or  disks. 

Acquisition  reform  movements  towards  commercial  practices  can  hopefully 
reduce  the  costs  of  operating  a  management  control  system,  while  still  providing  enough 
oversight  and  valid  data  as  to  not  negate  any  of  the  benefits  listed  in  table  eight. 

Conclusion 

In  this  chapter,  we  have  tried  to  synthesize  the  costs  and  benefits  of  the  criteria. 
However,  the  costs  we  have  identified  are  subjectively  based  upon  many  other  studies, 
and  the  benefits  are  equally  difficult  to  quantify.  Any  attempt  to  make  a  definitive 
conclusion  about  whether  the  benefits  outweigh  the  costs  or  vice  versa  would  be  overly 
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bold  and  virtually  defenseless.  Instead,  we  would  like  to  make  some  general  comments 
concerning  the  criteria. 

Many  other  major,  private  and  public,  non-military  defense  projects  have 
experienced  much  higher  cost  growth.  One  might  attribute  the  smaller  cost  growth  for 
the  military,  at  least  partially,  on  DoD’s  requirement  for  the  criteria  (which  were  not 
required  on  most  other  projects  identified  in  Table  2).  Based  on  Table  2,  DoD  projects 
experience  a  40%  cost  overrun  on  average.  On  the  other  hand,  the  average  overrun  of  the 
non-DoD  projects  identified  in  Figure  2  was  128%,  with  a  range  of  8%  to  593%  and  a 
median  of  70%.  Further,  Table  3  identifies  the  cost  of  C/SCSC  as  ranging  from  0.1%  - 
5%  of  total  contract  value.  Although  we  can  not  determine  if  the  difference  is  due  to 
C/SCSC  compliance,  or  other  factors,  it  does  indicate  that  0.1%  to  5%  of  total  contract 
value  is  a  small  price  to  pay  for  the  associated  C/SCSC  benefits. 

The  flexibility  of  the  criteria  allow,  and  even  encourage,  management  control 
systems  to  be  tailored  to  particular  decision  problems  and  contracts.  However,  we  must 
remember  that  there  are  many  customers  that  utilize  information  from  these  systems  for 
different  purposes.  A  requirement  for  excessively  reliable  information  may  result  in  less 
relevant,  and  more  costly  information  from  the  system.  Therefore,  tradeoffs  must  be 
made  to  ensure  these  systems  minimize  costs  while  maximizing  the  usefulness  of  the 
information  for  all  users. 


69 


V.  Conclusions/Limitations/F uture  Research 


Conclusions 

The  main  thrust  of  this  study  was  to  compare  the  costs  and  benefits  of  C/SCSC 
together  in  order  to  determine  the  utility  of  the  criteria.  Additionally,  we  wanted  to 
determine  how  the  change  to  EVMS  affects  these  costs  and  benefits.  Let  us  review  our 
original  research  questions  and  our  findings. 

Chapter  11  addresses  our  first  research  question:  What  are  the  costs  of  the  criteria? 
This  paper  examines  the  marginal  cost  of  C/SCSC,  which  is  the  difference  between  the 
cost  of  a  C/SC  SC-compliant  management  control  system  and  the  cost  of  a  contractor’s 
‘normal*  management  control  system.  Based  on  our  analysis,  we  conclude  that  the 
marginal  system  compliance  costs  of  a  C/SC  SC-compliant  system  ranges  from  334-481 
person  days.  Additionally,  we  conclude  that  the  marginal  system  operating  costs  are  50% 
of  the  C/SCSC-compliant  system  operating  costs.  These  cost  findings  are  consistent  with 
the  1 994  Coopers  &  Lybrand/TASC  study.  However,  where  the  Coopers  & 
Lybrand/TASC  study  limited  their  conclusions  by  only  addressing  the  costs  of  C/SCSC, 
we  have  addressed  both  the  costs  and  benefits  of  C/SCSC. 

Chapter  111  addresses  the  second  research  question:  what  are  the  benefits  of  the 
criteria?  Our  findings  related  to  the  benefits  of  the  criteria  are  summarized  in  Table  9. 
Again,  we  must  emphasize  the  fact  that  benefits  associated  with  earned  value  analysis  are 
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a  part  of  the  benefits  of  the  criteria.  Without  the  requirements  spelled  out  by  the  criteria, 
the  accuracy  and  the  utility  of  the  data  provided  by  such  a  system  would  be  very  suspect. 

Chapter  IV  addresses  our  third  research  question:  how  are  the  costs  and  benefits 
related?  In  comparing  our  identified  costs  with  our  identified  benefits,  a  subjective 
viewpoint  is  taken.  Because  of  the  difficulties  in  assigning  costs  to  the  benefits,  we  were 
not  able  to  conclude  that  the  benefits  outweigh  the  costs  or  vice  versa.  However,  we 
show  how  the  benefits  and  costs  are  related  in  terms  of  the  relevance  and  reliability  of 
cost  data  produced  by  the  contractor’s  cost  control  systems.  There  is  a  trade-off  which 
managers  make  between  relevance  and  reliability  that  affects  both  the  costs  and  benefits 
of  information  produced  by  a  C/SCSC  or  EVMS  compliant  management  control  system. 

Chapter  IV  also  addresses  the  fourth  and  final  research  question:  how  will  the 
move  to  EVMS  affect  these  costs  and  benefits?  The  change  to  EVMS  will  reduce  the 
contractor’s  system  certification  costs.  Other  costs  as  well  as  the  benefits  of  C/SCSC  will 
be  unchanged  as  a  result  of  the  switch  to  EVMS.  Also,  we  believe  EVMS  is  a  step  in  the 
right  direction,  but  the  impacts  of  this  move,  by  itself,  will  most  likely  result  in  only 
modest  improvements  over  C/SCSC. 


Limitations 

Although  this  study  was  an  attempt  to  objectively  compare  the  costs  and  benefits 
of  the  criteria,  there  are  limitations.  Many  of  the  major  studies  that  we  draw  upon  in  this 
thesis  were  based  on  questionnaires.  Questionnaires  reflect  individual  opinions  which  are 
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subjective.  However,  the  very  results  of  many  of  these  studies  would  seem  to  validate 
their  findings.  For  example,  we  would  expect  contractor’s  to  generally  oppose 
government  oversight  requirements.  Many  respondents  might  downplay  the  utility  of 
requirements  such  as  C/SCSC,  even  if  they  believed  them  to  be  critical  tools  for 
successful  management.  However,  many  of  the  studies  that  we  have  referenced  show 
significant  support  by  government  and  contractor  personnel  alike.  We  should  point  out 
that  many  of  these  same  supporters  acknowledged  that  there  were  some  burdensome 
areas  that  needed  improvement.  Many  of  the  problems  related  to  C/SCSC  are  associated 
with  its  implementation.  Better  training  and  other  steps  are  being  taken  to  help  minimize 
the  problems  associated  with  implementation.  The  switch  to  EVMS  and  other  acquisition 
reform  initiatives  should  also  help  ease  some  of  the  problems  experienced  in  the  past. 

Future  Research 

Any  effort  that  could  serve  to  lessen  the  burden  on  defense  contractors,  while  still 
providing  the  needed  oversight  mechanisms  for  government  personnel  would  certainly  be 
welcomed.  Any  such  efforts  would  probably  result  in  much  needed  cost  savings.  This 
need  is  greater  than  ever  before  because  of  steadily  declining  defense  budgets.  Most 
likely,  these  budgets  will  remain  at  levels  much  lower  than  experienced  in  the  past. 
However,  the  United  States  must  still  maintain  a  strong  military  force  to  counter  potential 
threats.  Thus,  the  adage  “do  more  with  less”  takes  on  even  greater  meaning.  The 
acquisition  of  major  military  weapon  systems  is  a  very  complicated  and  expensive 
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endeavor.  The  U.S.  defense  policy  has  focused  on  quality  weapon  systems  rather  than 
quantity  for  many  years.  Indeed,  we  have  enjoyed  technological  advantages  over  most  of 
our  adversaries.  However,  the  very  nature  of  these  cutting  edge  technology  acquisitions 
results  in  very  complex  weapon  systems.  The  complexities  of  these  systems,  coupled 
with  DoD’s  need  for  oversight,  have  resulted  in  a  very  cumbersome  acquisition  process. 
As  a  result,  many  reform  initiatives  have  been  promulgated  in  order  to  reduce  non-value- 
added  tasks  related  to  this  process  in  hopes  that  cost  savings  could  be  obtained.  At  the 
same  time,  government  representatives  must  have  the  proper  level  of  oversight  to  ensure 
that  needed  weapons  systems  are  delivered  on  time  to  meet  the  threat,  and  to  ensure  that 
public  funds  are  being  utilized  efficiently.  The  switch  to  EVMS  is  a  step  in  the  right 
direction,  but  we  must  be  careful  that  we  do  not  go  too  far  in  our  reform  efforts. 

Future  research  efforts  might  be  aimed  at  better  defining  the  optimal  level  of 
oversight  requirements.  We  believe  we  have  shown  how  C/SCSC  has  been  a  very 
effective  tool  in  the  past.  But,  could  there  be  any  new  modifications  to  the  criteria  that 
could  better  serve  the  process? 

Other  future  research  efforts  might  be  able  to  more  objectively,  in  an  empirical 
sense,  determine  the  costs  of  the  criteria.  Attempts  to  quantify  the  costs  of  C/SCSC  in  the 
past  have  been  very  subjective.  Today,  many  commercial  organizations  are  using  various 
earned  value  techniques  even  without  being  required  to  utilize  them.  One  solution  might 
be  to  find  a  private  firm  that  has  both  a  government  and  non-government  major  project 
that  utilize  earned  value  techniques  on  each.  If  such  a  firm  can  be  found,  and  the  two 
projects  are  somewhat  similar  in  scope  and  monetary  value,  then  a  comparison  could  be 
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made  to  determine  the  costs  on  the  government  contract  explicitly  related  to  criteria 
requirements. 

We  strongly  recommend  that  anyone  either  partaking  in  related  research  efforts  or 
looking  for  more  information  on  earned  value  reference  the  earned  value  homepage 
located  at  http://\wvw.acq.osd.mil/pm/. 
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Appendix  A:  A  Comparison  of  DoD  and  Industry  Criteria  (GAO  and  NSIA.  24-28) 


C/SCS  CRITERIA 

EVMS  CRITERIA 

Organization 

Organization 

1 .  Define  all  authorized  work  and  related 
resources  to  meet  the  requirements  of  the 
contract,  using  the  contract  work  breakdown 
structure  (WBS). 

1 .  Define  the  authorized  work  elements  for  the 
program.  A  work  breakdown  structure  (WBS), 
tailored  for  effective  internal  management 
control,  is  commonly  used  in  this  process. 

2.  Identify  the  internal  organizational  elements 
and  the  major  subcontractors  responsible  for 
accomplishing  the  authorized  work. 

2.  Identify  the  program  organizational  structure 
including  the  major  subcontractors  responsible 
for  accomplishing  the  authorized  work,  and  define 
the  organizational  elements  in  which  work  will  be 
planned  and  controlled. 

3.  Provide  for  the  integration  of  the  contractor’s 
planning,  scheduling,  budgeting,  work 
authorization  and  cost  accumulation  systems  with 
each  other,  the  contract  work  breakdown 
structure,  and  the  organizational  structure. 

3.  Provide  for  the  integration  of  the  company's 
planning,  scheduling,  budgeting,  work  authorization 
and  cost  accumulation  processes  with  each  other, 
and  as  appropriate,  the  program  work  breakdown 
structure  and  the  program  organizational  structure. 

4.  Identify  the  managerial  positions  responsible 
for  controlling  overhead  (indirect  costs). 

4.  Identify  the  company  organization  or  function 
responsible  for  controlling  overhead  (indirect  costs). 

5.  Provide  for  integration  of  the  contract  work 
breakdown  structure  with  the  contractor’s 
functional  organizational  structure  in  a  manner 
that  permits  cost  and  schedule  performance 
measurement  for  contract  work  breakdown 
structure  and  organizational  elements. 

5.  Provide  for  integration  of  the  program  work 
breakdown  structure  and  the  program  organizational 
structure  in  a  manner  that  permits  cost  and  schedule 
performance  measurement  by  elements  of  either  or 
both  structures  as  needed. 

Planning  and  Budgeting 

Planning,  Scheduling,  and  Budgeting 

6.  Schedule  the  authorized  work  in  a  manner  that 
describes  the  sequence  of  work  and  identifies  the 
significant  task  interdependencies  required  to 
meet  the  development,  production,  and  delivery 
requirements  of  the  contract 

6.  Schedule  the  authorized  work  in  a  manner  which 
describes  the  sequence  of  work  and  identifies 
significant  task  interdependencies  required  to  meet 
the  requirements  of  the  program. 

7.  Identify  physical  products,  milestones, 
technical  performance  goals,  or  other  indicators 
that  will  be  used  to  measure  output. 

7.  Identify  physical  products,  milestones, 
technical  performance  goals,  or  other  indicators 
that  will  be  used  to  measure  progress. 

8.  Establish  and  maintain  a  time-phased  budget 
baseline  at  the  cost  account  level  against  which 
contract  performance  can  be  measured.  Initial 
budgets  established  for  this  purpose  will  be 
based  on  the  negotiated  target  cost.  Any  other 
amount  used  for  performance  measurement 
purposes  must  be  formally  recognized  by  both 
the  contractor  and  the  Government. 

8.  Establish  and  maintain  a  time-phased  budget 
baseline,  at  the  control  account  level,  against 
which  program  performance  can  be  measured. 

Budget  for  far-term  efforts  may  be  held  in  higher 
level  accounts  until  an  appropriate  time  for 
allocation  at  the  control  account  level.  Initial 
budgets  established  for  performance 
measurement  will  be  based  on  either  internal 
management  goals  or  the  external  customer 
negotiated  target  cost  including  estimates  for 
authorized  but  undefinitized  work.  On  government 
contracts,  if  an  over  target  baseline  is  used  for 
performance  measurement  reporting  purposes,  prior 
notification  must  be  provided  to  the  customer. 
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9.  Establish  budgets  for  all  authorized  work  with 
separate  identification  of  cost  elements  (labor, 
material,  etc.). 

9.  Establish  budgets  for  authorized  work  with 
identification  of  significant  cost  elements  (labor, 
material,  etc.)  as  needed  for  internal  management 
and  for  control  of  subcontractors  . 

10.  To  the  extent  the  authorized  work  can  be 
identified  in  discrete,  short  span  work  packages, 
establish  budgets  for  this  work  in  terms  of 
dollars,  hours,  or  other  measurable  units.  Where 
the  entire  cost  account  can  not  be  subdivided 
into  detailed  work  packages,  identify  far  term 
effort  in  larger  planning  packages  for  budget  and 
scheduling  purposes 

10.  To  the  extent  it  is  practical  to  identify  the 
authorized  work  in  discrete  work  packages, 
establish  budgets  or  his  work  in  teims  of  dollars, 
hours,  or  other  measurable  units.  Where  the 
entire  control  account  is  not  subdivided  into 
work  packages,  identify  the  far  term  effort  in 
larger  planning  packages  for  budget  and 
scheduling  purposes. 

1 1 .  Provide  that  the  sum  of  all  work  package 
budgets,  plus  planning  package  budgets  within  a 
cost  account  equals  the  cost  account  budget. 

1 1 .  Provide  that  the  sum  of  all  work  package 
budgets  plus  planning  package  budgets  within  a 
control  account  equals  the  control 
account  budget. 

12.  Identify  relationships  of  budgets  or 
standards  in  work  authorization  systems  to 
budgets  for  work  packages. 

13.  Identify  and  control  level-of-effort  activity  by 
time-phased  budgets  established  for  this 
purpose.  Only  that  effort  which  cannot  be 
identified  as  discrete,  short  span  work  packages 
or  as  apportioned  effort  may  be  classed  as 
level-of-effort. 

12.  Identify  and  control  level  of  effort  activity  by 
time-phased  budgets  established  for  this 
purpose.  Only  that  effort  which  is  unmeasurable 
or  for  which  measurement  is  impractical  may  be 
classified  as  level  of  effort. 

14.  Establish  overhead  budgets  for  the  total  costs 
of  each  significant  organizational  component 
whose  expenses  will  become  indirect  costs. 

Reflect  in  the  contract  budgets  at  the  appropriate 
level  the  amounts  in  overhead  pools  that  are 
planned  to  be  allocated  to  the  contract  as 
indirect  costs. 

13.  Establish  overhead  budgets  for  each  significant 
organizational  component  of  the  company  for 
expenses  which  will  become  indirect  costs.  Reflect 
in  the  program  budgets,  at  the  appropriate  level,  the 
amounts  in  overhead  pools  that  are  planned  to  be 
allocated  to  the  program  as  indirect  costs. 

15.  Identify^  management  reserves  and 
undistributed  budget. 

14.  Identify  management  reserves  and 
undistributed  budget. 

16.  Provide  that  the  contract  target  cost  plus  the 
estimated  cost  of  authorized  but  unpriced  work 
is  reconciled  with  the  sum  of  all  internal  contract 
budgets  and  management  reserves 

15.  Provide  that  the  program  target  cost  goal  is 
reconciled  with  the  sum  of  all  internal  program 
budgets  and  management  reserves. 

Accounting 

Accounting  Considerations 

17.  Record  direct  costs  on  an  applied  or  other 
acceptable  basis  in  a  manner  consistent  with  the 
budgets  in  a  formal  system  that  is  controlled  by 
the  general  books  of  account. 

16.  Record  direct  costs  in  a  manner  consistent 
with  the  budgets  in  a  formal  system  controlled  by 
the  general  books  of  account. 
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1 8.  Summarize  direct  costs  from  cost  accounts 
into  the  work  breakdown  structure  without 
allocation  of  a  single  cost  account  to  two  or 
more  work  breakdown  structure  elements. 

1 7.  When  a  work  breakdown  structure  is  used, 
summarize  direct  costs  from  control  accounts 
into  the  work  breakdown  structure 
without  allocation  of  a  single  control  account  to 
two  or  more  work  breakdown  structure 
elements. 

19.  Summarize  direct  costs  from  the  cost  accounts 
into  the  contractor’s  functional  organizational 
elements  without  allocation  of  a  single  cost 
account  to  two  or  more  organizational  elements. 

18.  Summarize  direct  costs  from  the  control 
accounts  into  the  contractor's  organizational 
elements  without  allocation  of  a  single 
control  account  to  two  or  more  organizational 
elements. 

20.  Record  all  indirect  costs  which  will  be 
allocated  to  the  contract. 

19.  Record  all  indirect  costs  which  will  be 
allocated  to  the  contract. 

21 .  Identify  the  bases  for  allocating  the  cost  of 
apportioned  effort. 

22.  Identify  unit  costs,  equivalent  unit  costs,  or  lot 
costs  as  applicable. 

20.  Identify  unit  costs,  equivalent  units  costs,  or  lot 
costs  when  needed. 

23.  The  contractor’s  material  accounting  system 
will  provide  for: 

(1)  Accurate  cost  accumulation  and  assignment 
of  costs  to  cost  accounts  in  a  manner  consistent 
with  the  budgets  using  recognized,  acceptable 
costing  techniques. 

(2)  Determination  of  price  variances  by 
comparing  planned  versus  actual 
commitments. 

(3)  Cost  performance  measurement  at  the  point 
in  time  most  suitable  for  the  category  of  material 
involved,  but  no  earlier  than  the  time  of  actual 
receipt  of  material. 

(4)  Determination  of  cost  variances 
attributable  to  the  excess  usage  of  material. 

(5)  Determination  of  unit  or  lot  costs  when 
applicable. 

(6)  Full  accountability  for  all  material  purchased 
for  the  contract,  including  the  residual  inventory. 

21 .  For  EVMS,  the  material  accounting  system  will 
provide  for: 

(1)  Accurate  cost  accumulation  and  assignment 
of  costs  to  control  accounts  in  a  manner 
consistent  with  the  budgets  using 
recognized,  acceptable,  costing  techniques. 

(2)  Cost  performance  measurement  at  the  point 
in  time  most  suitable  for  the  category  of  material 
involved,  but  no  earlier  than  the 

time  of  progress  payments  or  actual  receipt  of 
material. 

(3)  Full  accountability  of  all  material  purchased 
for  the  program  including  the  residual  inventory. 
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Analysis 

Analysis  and  Management  Reports 

24.  Identify  at  the  cost  account  level  on  a  monthly 
basis  using  data  from,  or  reconcilable  with,  the 
accounting  system: 

(1)  Comparison  of  budgeted  cost  for  work 
scheduled  and  budgeted  cost  of  work 
performed; 

(2)  Comparison  of  budgeted  cost  for  work 
performed  and  actual  (applied  where 
appropriate)  direct  costs  for  the  same  work;  and 

(3)  Variances  resulting  from  the  comparisons 
between  the  budgeted  cost  for  work  scheduled 
and  the  budgeted  cost  for  work  performed  and 
between  the  budgeted  cost  for  work  performed 
and  actual  or  applied  direct  costs,  classified  in 
terms  of  labor,  material,  or  other  appropriate 
elements  together  with  the  reasons  for  significant 
variances. 

22.  At  least  on  a  monthly  basis,  generate  the 
following  information  at  the  control  account  and 
other  levels  as  necessary  for  management  control 
using  actual  cost  data  from,  or  reconcilable  with, 
the  accounting  system: 

(1)  Comparison  of  the  amount  of  planned  budget 
and  the  amount  of  budget  earned  for  work 
accomplished.  This  comparison 

provides  the  schedule  variance. 

(2)  Comparison  of  the  amount  of  the  budget 
earned  the  actual  (applied  where  appropriate) 
direct  costs  for  the  same  work.  This 
comparison  provides  the  cost  variance. 

25.  Identify  on  a  monthly  basis,  in  the  detail 
needed  by  management  for  effective  control, 
budgeted  indirect  costs,  actual  indirect  costs,  and 
cost  variances  with  the  reasons  for  significant 
variances. 

23.  Identify  budgeted  and  applied  (or  actual) 
indirect  costs  at  the  level  and  frequency  needed 
by  management  for  effective  control, 
along  with  the  reasons  for  any  significant 
variances 

26.  Summarize  the  data  elements  and  associated 
variances  listed  in  subparagraphs  a.(l)  and  (2), 
directly  above,  through  the  contractor 
organization  and  work  breakdown  structure  to 
the  reporting  level  specified  in  the  contract. 

24.  Summarize  the  data  elements  and  associated 
variances  through  the  program  organization 
and/or  work  breakdown  structure  to 
support  management  needs  and  any  customer 
reporting  specified  in  the  contract. 

27.  Identify  significant  differences  on  a  monthly 
basis  between  planned  and  actual  schedule 
accomplishment  and  the  reasons. 

25.  Identify,  at  least  monthly,  the  significant 
differences  between  both  planned  and  actual 
schedule  performance  and  planned  and 
actual  cost  performance,  and  provide  the 
reasons  for  the  variances  in  the  detail  needed  by 
program  management. 

28.  Identify  managerial  actions  taken  as  a  result  of 
criteria  items  in  paragraphs  a.  through  d.,  directly 
above. 

26.  Implement  managerial  actions  taken  as  the 
result  of  earned  value  information. 

29.  Based  on  performance  to  date,  on  commitment 
values  for  material,  and  on  estimates  of  future 
conditions,  develop  revised  estimates  of  cost  at 
completion  for  work  breakdown  structure 
elements  identified  in  the  contract  and  compare 
these  with  the  contract  budget  base  and  the  latest 
statement  of  funds  requirements  reported  to  the 
Government. 

27.  Develop  revised  estimates  of  cost  at 
completion  based  on  performance  to  date, 
commitment  values  for  material,  and  estimates 
of  future  conditions.  Compare  this  information 
with  the  performance  measurement  baseline  to 
identify  variances  at  completion  important  to 
company  management  and  any  applicable 
customer  reporting  requirements  including 
statements  of  funding  requirements. 
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Revisions  and  Access  to  Data 

Revisions  and  Data  Maintenance 

30.  Incorporate  contractual  changes  expeditiously, 
recording  the  effects  of  such  changes  in  budgets 
and  schedules,  in  the  directed  effort  prior  to 
negotiation  of  a  change,  base  such  revisions  on 
the  amount  estimated  and  budgeted  to  the 
functional  organizations. 

28.  Incorporate  authorized  changes  in  a  timely 
manner,  recording  the  effects  of  such  changes  in 
budgets  and  schedules.  In  the 
directed  effort  prior  to  negotiation  of  a  change, 
base  such  revisions  on  the  amount  estimated  and 
budgeted  to  the  program  organizations. 

3 1 .  Reconcile  original  budgets  for  those  elements 
of  the  work  breakdown  structure  identified  as 
priced  line  items  in  the  contract,  and  for  those 
elements  at  the  lowest  level  in  the  program  work 
breakdown  structure,  with  current  performance 
measurement  budgets  in  terms  of  changes  to  the 
authorized  work  and  internal  replanning  in  the 
detail  needed  by  management  for  effective 
control. 

29.  Reconcile  current  budgets  to  prior  budgets  in 
terms  of  changes  to  the  authorized  work  and 
internal  replanning  in  the  detail 
needed  by  management  for  effective  control. 

32.  Prohibit  retroactive  changes  to  records 
pertaining  to  work  performed  that  would  change 
previously  reported  amounts  for  direct  costs, 
indirect  costs,  or  budgets,  except  for  correction 
of  errors  and  routine  accounting  adjustments. 

30,  Control  retroactive  changes  to  records 
pertaining  to  work  performed  that  would  change 
previously  reported  amounts  for  actual  costs, 
earned  value,  or  budgets.  Adjustments  should  be 
made  only  for  correction  of  errors,  routine 
accounting  adjustments,  effects  of  customer  or 
management  directed  changes,  or  to  improve  the 

1  baseline  integrity  and  accuracy  of  performance 
measurement  data. 

33.  Prevent  revisions  to  the  contract  budget  base 
except  for  Government-directed  changes  to 
contractual  effort. 

3 1 .  Prevent  revisions  to  the  program  budget 
except  for  authorized  changes. 

34.  Document  internally  the  changes  to  the 
performance  measurement  baseline  and  notify 
expeditiously  the  procuring  activity  through 
prescribed  procedures. 

32.  Document  changes  to  the  performance 
measurement  baseline. 

35.  Provide  the  Contracting  Officer  and  the 
Contracting  Officer's  authorized 
representatives  with  access  to  the 
information  and  supporting  documentation 
necessary  to  demonstrate  compliance  with 
the  cost/schedule  control  systems  criteria. 
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Appendix  B;  Abstracts  of  Previous  C/SCSC  Cost  and  Benefit  References 

Title:  Preface  to  C/SCSC  White  Paper 
Author:  Abba,  Wayne 
Format:  Article 

Journal:  Program  Manager  Volume  15 
Date:  November-December  1986 

This  paper  summarizes  the  objectives  of  C/SCSC  and  the  associated  cost 
performance  reports,  as  well  as  what  can  and  cannot  be  expected  from  their  use.  First,  the 
objectives  and  requirements  of  C/SCSC  are  given.  Next,  the  article  points  out  the 
necessity  of  establishing  a  baseline  and  how  it  interacts  with  earned  value  management. 
There  is  a  brief  discussion  of  how  the  government  validates  the  contractor’s  management 
control  system  to  ensure  it  complies  with  the  C/SCSC  requirements.  There  is  also  a  brief 
narrative  about  the  Cost  Performance  Report  (CPR)-the  summar>'  report  provided  to  the 
government  from  the  contractor’s  approved  management  control  system. 

Next,  the  author  talks  about  challenges  such  as  implementation  time  and  system 
discipline  requirements  that  result  from  implementing  C/SCSC.  Although  not  a  direct 
result  of  the  criteria,  these  issues  can  affect  the  cost  of  implementing  and  operating  a 
C/SCSC  compliant  management  system.  The  author  highlights  the  fact  that  the  costs  of 
C/SCSC,  which  some  proclaim  to  be  excessive,  are  extremely  difficult  to  quantify.  It  is 
almost  impossible  to  separate  the  additional  costs  of  C/SCSC  from  the  management  costs 
that  would  be  incurred  anyway.  Some  reasons  for  improper  implementation  are  given 
along  with  a  brief  discussion  of  the  Cost/Schedule  Status  Report  (C/SSR). 
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Finally,  the  paper  asserts  that  a  C/SCSC-compliant  system  is  the  best  tool 
available  to  assure  contractor  use  adequate  cost  and  schedule  management  control 
systems.  This  will  ensure  that  valid  cost  and  schedule  performance  data  are  generated 
allowing  managers  to  make  good  business  decisions. 
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Title:  Cost  Growth  and  the  Use  of  Competitive  Acquisition  Strategies 
Author:  Frederick  P.  Biery 
Format:  Article 

Date/Published:  Presented  at  the  1985  National  Estimating  Society  Conference 
Pages:  11-17 

This  article  was  an  attempt  to  quantify  the  costs  and  benefits  of  competition  in 
acquisition  strategies.  It’s  no  secret  that  the  military  has  experienced  cost  growth  on  most 
of  its  contracts.  In  fact,  the  Pentagon  has  attracted  much  criticism  as  a  result  of  program 
cost  growth.  However,  the  author  hints  that  the  military’s  cost  growth  may  not  look  so 
bad  when  compared  with  other  projects.  A  table  is  provided  summarizing  cost  growth 
experienced  by  complex,  technically  ambitious  non-military  projects.  Indeed,  many 
public  sector  projects  and  private  sector  projects  exhibited  greater  (sometimes 
significantly  greater)  cost  growth  than  the  military.  The  author  points  to  the  variety  of 
management  initiatives  put  into  place  in  the  early  1970s  which  may  have  had  some 
positive  impacts  upon  improving  cost  and  schedule  controls  within  the  military.  Several 
policies  under  the  Packard  initiatives  are  mentioned  as  well  as  the  creation  of  the  Cost 
Analysis  Improvement  Group.  However,  the  author  does  not  mention  specifically  the 
possibility  that  earned  value  techniques  within  the  Cost/Schedule  Control  Systems 
Criteria  (C/SCSC)  could  have  played  a  part  in  helping  to  minimize  cost  growth. 
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Title:  The  Cost  of  the  “Criteria” 


Author:  Richard  Brodkorb,  Decision  Planning  Corporation 
Format:  Report 
Date/Published:  11  FEB  1992 
Pages:  14 

The  purpose  of  this  study  is  to  determine  what  it  costs  to  design  and  implement  a 
management  control  system  which  is  compliant  with  the  C/SCSC.  The  main  difficulty  in 
determining  the  true  cost  of  C/SCSC  is  separating  the  normal  program  control  costs  of 
good  program  management  from  the  cost  of  a  C/SCSC  compliant  system.  The  study 
separates  the  installation  of  C/SCSC  compliant  system  into  five  areas:  1)  system  design, 
2)  system  documentation,  3)  training,  4)  implementation,  and  5)  system  evaluation. 

This  study  assumes  a  generic  R&D  contract  of  $75M,  a  contract  duration  of  three 
years,  and  80  cost  accounts  with  40  cost  account  managers.  Additionally,  the  study 
assumes  it  takes  twelve  months  to  achieve  a  validated  system. 

System  design  costs  require  the  full-time  of  a  three  or  four  person  team  for 
approximately  four  months.  Additionally,  a  steering  committee  of  senior  managers  will 
aid  in  the  review  of  the  system  design.  This  study  estimates  that  the  system  design  effort, 
given  the  assumptions,  will  take  240  person  days.  The  study  concludes  that  the  system 
design  effort  does  not  significantly  vary  with  contract  size.  However,  if  large  complex 
existing  cost  control  systems  are  in  place,  this  effort  could  increase. 
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During  system  documentation  a  system  description  and  detailed  operating 
procedures  are  developed.  This  study  estimates  system  documentation  to  take  76  person 
days,  which  is  also  significantly  independent  of  contract  size. 

Training  involves  the  cost  account  managers,  financial  personnel,  functional 
managers,  and  senior  executives.  The  study  indicates  that  cost  account  managers 
generally  receive  20  to  40  hours  of  training,  while  other  personnel  receive  4  to  16  hours 
of  training.  The  study  estimated  the  total  training  effort  to  take  213  person  days.  This 
estimate  varies  directly  with  the  size  of  the  contract,  as  larger  contracts  require  more  cost 
account  managers  and  other  personnel. 

The  largest  costs  associated  with  implementing  a  C/SCSC  compliant  system  is 
system  implementation.  Unfortunately,  these  costs  are  the  most  difficult  to  isolate  from 
normal  program  management  controls.  This  effort  consists  of  the  following  activities: 
preparing  the  CWBS  dictionary,  preparing  the  cost  account  (CA)  work  authorization 
documents,  preparing  CA  plans,  reviewing  CA  plans,  establishing  the  baseline  log, 
opening  charge  numbers,  analyzing  cost  account  variances,  preparing  cost  performance 
reports,  and  statusing  cost  account  plans.  The  effort  is  estimated  at  493  person  days.  An 
important  note:  this  estimate  is  only  for  the  twelve  months  prior  to  validation,  further 
implementation  costs  will  be  incurred  throughout  the  contract.  This  effort  will  also  vary 
directly  with  contract  size. 

System  evaluation  includes  three  major  Government  reviews  (Implementation 
review,  readiness  review,  and  demonstration  review).  These  formal  reviews  can  also  be 
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supplemented  with  less  formal  visits,  and  contractor  mock  reviews.  This  effort  is 
estimated  at  175  person  days,  and  will  vary  directly  with  contract  size. 

In  addition  to  the  costs  in  these  five  areas,  there  are  support  costs  that  are 
addressed.  Administrative  and  reproduction-related  efforts  are  estimated  at  300  person 
days,  and  $12,000  of  supplies.  C/SCSC  compliant  systems  may  be  automated,  but  the 
costs  of  this  are  not  included  in  this  study. 

This  study  concluded  that  the  total  pre-validation  effort  is  1 ,497  person  days,  and 
an  additional  $12,000  of  supplies.  This  equates  to  6.24  person  years,  and  assuming  a 
burdened  labor  rate  of  $50/hr,  $610,800.  Any  automation  costs  will  add  to  this  total.  The 
study  does  note  that  doubling  the  contract  size  (to  $150M),  only  increases  the  costs  by 
30%. 
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Title:  The  DoD  Regulatory  Cost  Premium:  A  Quantitative  Assessment 

Author:  Coopers&Lybrand/TASC 

Format:  Deputy  Secretary  of  Defense  Requested  Study 

Date/Published:  December  1994 

Pages:  70 

DoD  and  Congress  have  developed  a  complex  regulatory  environment  intended  to 
maintain  public  accountability  in  defense  acquisition  and  prevent  contractor  abuses.  The 
need  to  protect  against  contracting  abuses  must  be  balanced  by  a  concern  for  the  cost  of 
this  oversight.  Many  defense  contractors  now  find  that  their  core  capability  often  has 
relatively  little  to  do  with  technical  capability  or  efficiency,  and  a  lot  to  do  with  their 
ability  to  deal  with  the  government  and  comply  with  its  requirements.  The  purpose  of 
this  study  is  to  assist  DoD’s  reform  efforts  by  providing  credible,  empirically-based 
estimates  of  the  industry  costs  impact  of  DoD  regulation  and  oversight. 

Specifically,  this  study  measured  value  added  costs  which  are  equal  to  total  costs 
minus  material  costs.  To  determine  the  DoD  cost  premium,  cost  center  managers  were 
asked  to  estimate  the  cost  impact  on  specific  activities  of  substituting  the  best  commercial 
practices  for  all  DoD  regulation  and  oversight.  To  simplify: 


DoD  Cost  Premium 

Actual  Cost  of 

Activity  Cost  Without 

Activity 

DoD  Regs  &  Oversight 
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The  DOD  cost  premium  percentage  is  then  equal  to  the  contractor  compliance 
costs  divided  by  the  value  added  costs.  In  this  study,  C/SCSC  compliance  ranked  as  the 
third  highest  DoD  cost  premium.  The  study  cites  that  C/SCSC  accounts  for  0.9%  of  total 
value  added  costs,  and  5.1%  of  the  total  cost  premium.  Compliance  costs  were  collected 
for  both  C/SCSC  costs,  and  C/SSR  costs.  The  study  notes  that  C/SCSC  control 
requirements  have  a  major  cost  impacts  on  two  sites,  and  are  significant  at  three  other 
sites.  Program  managers  devote  a  significant  share  of  their  time  performing  variance 
analysis  and  preparing  the  reports.  Finance  personnel  monitor  reporting  compliance, 
develop  and  maintain  automated  reporting  system,  generate  and  format  the  budget  vs. 
Cost  data  in  the  periodic  reports,  and  resolve  related  progress  payment  issues  with  the 
government. 

It  is  important  to  note  that  this  study  did  not  address  the  associated  benefits  that 
result  from  the  costs  of  C/SCSC  compliance.  In  determining  the  usefulness  of  C/SCSC, 
both  the  costs  and  benefits  must  be  addressed.  Every  commodity,  including  information, 
has  associated  costs.  Only  after  looking  at  the  benefits  of  said  commodity  can  one 
determine  if  the  commodity  is  useful.  Although  this  study  did  not  address  the  benefits,  it 
did  look  at  the  contractors’  perceptions  of  C/SCSC. 

In  general,  industry  views  the  framework  and  principles  of  C/SCSC  positively. 
Sound  program  management  requires  regular  analysis  of  expenditure  and  performance 
trends,  and  deviations  must  be  addressed  early  to  avoid  overruns  and  delays.  However, 
all  contractors  subject  to  C/SCSC  agree  that  reporting  is  too  detailed,  repetitive,  and 
voluminous  to  be  used  effectively  as  a  management  tool  by  either  the  government  or 
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industr>’,  and  that  the  requirement  may  undermine  program  performance  by  diverting  the 
time  and  attention  of  the  program  manager.  Industry  suggests  that  DoD  require  reporting 
to  summary  levels  in  the  WBS  structure  only;  limit  variance  analysis  to  only  those 
deviations  that  truly  warrant  analysis  and  corrective  actions;  or  discard  C/SCSC  and 
replace  with  standardized  commercial  cost  and  schedule  reporting. 
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Title:  Cost/Schedule  Control  Systems  Criteria:  It  may  be  more  than  we  think 
Author:  Irwin  J.  Faibisch 
Format:  Article 

Date/Published:  Program  Manager:  September/October  1991 
Pages:  30-33 

C/SCSC  are  often  defined  as  criteria  which,  when  utilized  properly,  integrate  cost 
schedule  and  technical  performance  into  a  singular  management  control  system  to 
promote  better  overall  management  of  a  project.  The  author  discusses  how  the  technical 
performance  aspect  has  lost  some  of  its  meaning  in  the  definition.  He  states  that 
C/SCSC’s  principal  value  lies  in  its  program  control  and  planning  attributes.  Although 
the  author  believes  reports  generated  by  a  C/SCSC  system  are  important,  he  believes  that 
too  much  emphasis  is  placed  on  them.  They  are  merely  after  the  fact  reports  that  tell  us 
where  we’ve  been  and  are  more  reactive  in  nature.  He  strongly  advocates  the  need  to  use 
C/SCSC  internally,  in  real-time,  so  that  managers  can  take  appropriate  action  at  the  most 
opportune  time-as  problems  occur.  In  this  way,  they  can  best  be  tied  to  technical 
accomplishment  indicators. 
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Title:  FEEDBACK  The  Cost  of  C/SCSC 

Author: 

Format:  A  letter  received  by  a  DSMC  faculty  member  in  the  Program  Managers 
Newsletter 

Date/Published:  Jan-Feb  1978 
Pages: 

This  letter  discusses  the  cost  avoidance  due  to  improved  discipline  and 
management  visibility  from  the  implementation  of  C/SCSC.  The  government  had 
decided  to  delete  a  scheduled  task  from  the  contract  and  request  a  credit  for  the  reduction 
in  scope.  Since  none  of  the  work  being  deleted  had  been  started,  the  only  obstacle  was 
determining  the  value  of  the  work. 

The  work  had  been  planned  in  the  contractors  C/SCSC  system,  so  preparation  of 
the  proposal  was  almost  effortless.  Additionally,  government  review  of  the  proposal  was 
also  simplified.  The  contractor  and  government  were  less  than  2%  apart  at  the  start  of 
negotiations,  and  settled  on  a  price  in  minutes. 

The  savings  were  quantified  by  assessing  the  proposal  preparation,  fact  finding, 
and  negotiation  times.  Total  cost  of  proposal  preparation  was  only  $200,  compared  to 
past  proposal  costs  of  $1 0,000.  The  government  used  only  eight  person  hours  for  fact 
finding  versus  the  usual  100.  Negotiations  usually  take  100  person  hours,  so  total 
government  savings  (fact  finding  and  negotiations)  is  estimated  at  $5,000.  The  total 
savings  was  estimated  to  be  approximately  $15,000. 

In  another  case,  a  program  with  two  competing  developers  was  canceled.  The 
associated  problem  was  in  determining  the  target  cost  which  the  contractors’  fees  would 
be  based  on.  Using  reports  and  planning  from  the  contractor’s  C/SCSC  validated  system. 
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the  project  office  and  one  contractor  reached  agreement  instantaneously,  while  it  took 
three  days  for  the  other  contractor.  The  immediate  savings  to  the  government  was 
$400,000,  while  the  manpower  savings  may  have  been  greater  since  a  claims  situation 
was  avoided. 
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Title:  Earned  value  project  management 
Author:  Fleming,  Q.  W.,  &  Hoppleman,  J.  M. 

Format:  Book 

Published:  Upper  Darby  Pennsylvania:  Project  Management  Institute,  1996 
Total  Pages:  141 

In  this  book,  the  authors  provide  a  very  simple,  yet  quite  comprehensive 
discussion  of  the  earned  value  concept.  The  first  part  of  the  book  describes  what  earned 
value  means,  why  it  came  about  and  how  it  has  evolved.  The  next  part  of  the  book  details 
how  to  implement  earned  value  and  use  it  to  manage  more  effectively.  The  last  chapter 
of  the  book  discusses  a  simplified  method  of  using  earned  value  for  private  sector 
applications. 

Chapter  three  is  especially  useful  for  our  thesis  effort.  Here  the  authors  list  ten  of 
the  most  important  contributions  resulting  from  the  employment  of  earned  value  based  on 
their  review  of  the  existing  body  of  knowledge  on  the  subject.  A  brief  summary  of  these 
benefits  are  described  below. 

1 .  The  employment  of  a  single  management  control  system  providing  accurate, 

consistent,  reliable,  and  timely  data  that  management  at  all  levels  can  use  to  monitor 
performance  throughout  the  life  cycle  of  a  project  or  a  repetitive  production  effort. 

The  authors  point  out  the  problems  encountered  in  the  past  resulting  from  firms 
keeping  multiple  sets  of  books.  Data  provided  to  project  managers,  senior  management, 
financial  analysts  and  others  often  was  inconsistent.  Fortunately,  earned  value 
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management  standards  eliminates  this  problem  by  requiring  one  system  as  outlined 
above. 

2.  A  management  approach  that  integrates  the  technical  scope  of  work,  the  schedules, 
and  the  costs,  allowing  for  the  continuous  measurement  of  integrated  performance 
throughout  the  life  cycle  of  the  project  or  a  production  effort. 

The  advent  of  the  Work  Breakdown  Structure  (WBS)  in  the  early  1960s  gave  us  the 
ability  to  integrate  cost,  technical  and  schedule  aspects  of  a  project.  Instead  of  projects 
being  defined  and  measured  by  parochial  interests  in  a  non-integrated  manner  as  was 
frequently  the  case  in  the  past,  now  performance  measurement  could  take  place  in  a  much 
more  highly  integrated  manner.  This  new  integrated  approach  served  to  help  various 
functional  personnel  speak  a  common  project  language  as  never  before.  Further,  various 
functional  metrics  became  much  easier  to  relate  to  each  other. 

3.  Documented  empirical  data  collected  from  over  700  DOD  contracts. .  .reflecting  a 
pattern  of  consistent  and  predictable  performance  history  (Beach  1 990). 

The  analysis  of  these  contracts  that  employ  earned  value  has  provided  some  important 
insights  about  government  contracts.  Sadly  enough,  after  a  project  is  about  15% 
complete,  history  has  shown  there  is  very  little  chance  that  any  overruns  can  be 
overcome.  In  fact,  studies  show  that  the  overruns  usually  will  get  worse.  We  can 
however,  usually  predict  the  final  cost  and  schedule  this  early  in  a  project  within  a 
predictable  range  of  values  based  on  historical  performance. 

4.  The  utility  and  the  stability  of  the  cumulative  Cost  Performance  Index-CPI(e)  based 
on  cost  efficiencies  to  continuously  monitor  the  performance  trends  of  a  project. 


93 


Again,  using  cumulative  contract  performance  data,  management  is  able  to  assess  the 
performance  to  date  and  to  predict  the  final  performance  results  quite  well  at  a  very  early 
point. 

5.  The  utility  of  the  Schedule  Performance  Index  (SPI)  to  monitor  and  quantify  the  value 
of  the  work  scheduled  and  to  compare  it  against  the  value  of  work  physically 
performed. 

A  contractor  that  uses  earned  value  management  standards  can  easily  compute  this 
index  which  allows  us  to  determine  how  far  ahead  or  behind  the  planned  schedule  we  are. 
The  SPI  can  be  used  in  conjunction  with  the  Critical  Path  Method  (CPM)  to  help  manage 
the  project  schedule  more  effectively. 

6.  The  utility  of  the  cumulative  Cost  Performance  Index(e)  to  statistically  forecast  a 
‘iow-end”  range  of  estimated  costs  at  completion. 

A  contractor  that  uses  earned  value  management  standards  correctly  can  also  compute 
the  cumulative  CPI  quite  easily.  This  index  provides  what  many  consider  to  be  the  best 
case  or  “low  end”  amount  of  funds  needed  to  finish  the  effort.  This  index  has  proven  to 
be  a  very  stable  indicator  of  performance. 

7.  The  utility  of  the  cumulative  Cost  Performance  Index(e)  combined  with  the  Schedule 
Performance  Index  to  statistically  forecast  the  “most  likely”  range  of  estimated  costs 
at  completion. 

Combining  both  indexes  allows  us  to  account  for  the  compounding  effects  of 
performance  that  is  both  behind  schedule  and  over  cost.  This  method  is  considered  by 
some  to  be  a  good  method  to  estimate  “most  likely”  final  cost  results. 
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8.  The  utility  of  the  To-Complete  Performance  Index  (TCPI)  to  monitor  the  remaining 


project  tasks  against  a  specific  management  financial  goal. 

Basically,  this  index  is  calculated  by  dividing  the  remaining  work  by  the  remaining 
budget  (or  a  revised  budget  goal).  The  resulting  index  shows  what  level  of  performance 
must  be  attained  to  successfully  reach  this  goal. 

9.  The  utility  of  a  weekly  (or  periodic)  Cost  Performance  Index-CPI(p)  based  on  the 
actual  performance  it  took  to  achieve  a  planned  production  standard. 

A  periodic  CPI  can  be  used  by  industrial  engineers  or  others  to  track  the  cost  performance 
achieved  against  an  established  production  standard  for  a  given  period. 

10.  The  use  of  Management  by  Exception  (MBE)  principles  to  focus  management’s 
attention  on  significant  exceptions  to  the  authorized  plan,  thus  allowing  management 
to  effectively  monitor  all  critical  aspects  of  performance  and  then  develop  and  apply 
timely  corrective  actions. 

Obviously,  it  is  a  major  benefit  for  management  to  be  able  to  oversee  the 
performance  of  a  project  without  having  to  delve  into  minute  details. 
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Title:  The  Cost  of  C/SCSC 


Author:  Gadeken,  Owen  C.  and  Tison,  Thomas  S. 

Format:  Article  in  Program  Manager 
Date/Published:  July-August  1983 
Pages:  6 

The  purpose  of  this  study  is  to  analyze  the  cost  impact  of  implementing  C/SCSC. 
The  study  approaches  this  by  analyzing  the  effect  of  implementing  C/SCSC  on  a 
hypothetical,  well-managed  company. 

The  authors  take  the  approach  that  the  use  of  C/SCSC  is  very  similar  to  the  use  of 
engineering  specifications  and  design  criteria.  Contractors  build  their  widgets  in 
accordance  with  the  specifications,  but  have  the  flexibility  to  propose  any  design  such 
that  it  meets  the  specifications.  They  emphasis  that  the  C/SCSC  are  not  a  management 
system,  but  rather  standards  for  evaluating  contractor  management  control  systems.  An 
interesting  note  is  that  the  C/SCSC  do  not  require  the  submission  of  any  reports,  but 
specify  the  reporting  capabilities  that  the  contractors’  systems  must  have.  The  type  and 
detail  of  reports  is  then  selected  by  the  program  office. 

The  major  difficulty  in  assessing  the  cost  of  C/SCSC  is  identifying  costs  uniquely 
attributable  to  C/SCSC.  Validation  reviews  and  system  documentation  are  clearly  related 
to  C/SCSC.  However,  planing,  budgeting,  variance  analysis,  and  other  daily  activities  are 
difficult  to  separate  from  normal  management  processes.  Other  complicating  factors 
include  the  modification  effort  of  the  contractor’s  existing  system,  the  government 
review  team’s  standards,  and  the  attitude  of  the  contractor  when  it  reports  C/SCSC  costs. 
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Past  studies  indicate  that  contractor  bias  plays  a  role  in  determining  the  costs  attributed  to 
C/SCSC. 

To  determine  the  cost  of  C/SCSC,  this  study  moves  to  the  viewpoint  of  a 
hypothetical  well-managed  company  with  the  following  assumption:  the  company  has  a 
completely  integrated  management  and  control  system.  The  authors  identify  many 
weaknesses  associated  with  their  approach.  First,  there  is  no  such  thing  as  an  ideal,  well- 
managed  company.  Next,  the  analysis  is  highly  subjective.  Finally,  the  costs  or  range  of 
costs  of  implementing  C/SCSC  are  not  quantified. 

The  study  begins  its  analysis  by  evaluating  the  company  in  the  five  criteria  areas 
of  C/SCSC.  The  organizational  criteria  require  the  contractor  to  define  all  of  the  work  to 
be  performed  under  the  contract,  and  to  assign  this  work,  using  a  CWBS,  to  organizations 
responsible  for  it.  These  criteria  represent  basic  management  principles  that  any  well- 
managed  company  can  meet  with  minimal  additional  effort. 

The  planning  and  budget  criteria  require  all  authorized  work  to  be  planned, 
scheduled,  budgeted,  and  authorized.  Sound  management  principles  suggest  that  tasks 
must  be  planned  and  scheduled,  assigned  budgets,  and  authorized  work.  These  principles 
should  also  be  found  in  a  well-managed  company.  Implied  in  these  criteria  are  planning 
at  levels  of  detail  that  may  or  may  not  be  found  in  a  well-managed  company.  The 
planning  down  to  cost  accounts  and  work  packages  may  be  effort  not  usually  performed 
by  a  well-managed  company. 

The  accounting  criteria  require  that  the  contractor’s  accounting  system  record  all 
direct  and  indirect  costs  applicable  to  the  contract.  Contractors  must  be  able  to 
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summarize  costs  from  the  level  that  they  are  applied  up  through  the  WBS  and  functional 
organizational  structure.  Except  for  one,  these  criteria  are  essentially  those  required  by 
the  Defense  Contract  Audit  Agency  (DCAA)  for  all  defense  contractors.  The  one 
criterion  not  required  by  the  DCAA  deals  with  material  costs,  and  requires  most 
companies  to  add  practices  and  procedures  to  comply  with  C/SCSC. 

The  analysis  criteria  establish  the  characteristics  that  contractors’  systems  must 
posses,  and  specify  the  type  of  data  that  should  be  derived  from  the  contractors’  systems 
to  adequately  measure  and  address  performance.  In  essence,  these  criteria  require  a 
comparison  of  actual  versus  planned  performance,  calculation  of  variances,  and  analysis 
of  N'ariances.  The  study  states  that  most  well-managed  companies  do  not  have  formal 
methods  of  determining  earned  value  established. 

The  revisions  and  access  to  data  criteria  require  the  contractor  to  maintain  a  valid 
performance  measurement  baseline,  and  provide  government  access  to  internal  data. 

Most  well-managed  companies  do  not  have  procedures  to  meet  these  criteria,  however, 
compliance  to  them  would  not  take  much  effort. 

The  validation  of  compliance  to  C/SCSC  is  usually  completed  after  reviews  by 
government  teams.  The  teams  usually  consist  of  seven  to  fifteen  people.  These  reviews 
require  additional  effort  from  both  the  contractor  and  government. 

This  study  concludes  that  there  are  cost  impacts  associated  with  imposing 
C/SCSC,  but  it  is  impossible  to  quantify  these  costs.  It  is,  however,  possible  to  identify 
the  major  cost  drivers  and  their  sources.  The  major  cost  drivers  are  the  implementation  of 
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earned  value,  and  the  level  of  detail  and  rigid  interpretation.  The  minor  cost  drivers  are 
the  terminology,  material  accountability,  and  change  control. 

This  study  recommends  several  ways  of  minimizing  the  cost  drivers.  First,  the  DoD  must 
ensure  knowledgeable  and  experienced  C/SCSC  team  leaders  and  members  are 
appointed.  The  team  leaders  and  members  must  work  with  the  contractor,  specifically  in 
the  implementation  process,  to  help  the  contractor  fully  understand  the  C/SCSC 
requirements.  Also,  the  DoD  must  ensure  that  all  members  of  the  government  review 
team  are  properly  trained.  Several  courses  are  available  to  DoD  employees  which  can 
provide  adequate  training. 
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Title:  Major  Acquisitions:  Significant  Changes  Underway  in  DOD’s  Earned  Value 
Management  Process 

Author: 

Format:  GAO  Report 
Date/Published:  May  1997 
Pages:  1-31 

This  report  addresses  the  problems  facing  the  Cost/Schedule  Control  System 
Criteria  (C/SCSC)  process.  It  also  summarizes  the  progress  resulting  from  DoD  reforms 
and  some  challenges  that  DoD  will  face  during  the  implementation  and  management  of 
these  changes.  The  evolution  of  earned  value  is  given  from  the  inception  of  C/SCSC 
through  the  current  Industry  Standard  Guidelines  for  Earned  Value  Management  Systems 
(EVMS). 

As  further  testament  to  the  value  of  earned  value,  the  report  highlights  the  fact 
that  commercial  firms  are  adopting  the  concept  of  earned  value  more  and  more. 

Although  many  of  these  new  implementations  of  C/SCSC  are  being  applied  in  a  more 
streamlined  manner  than  in  the  past,  the  basic  concept  of  earned  value  is  still  very  much 
alive  and  well.  For  example,  United  Defense  Limited  Partnership  (a  major  defense 
contractor)  has  incorporated  earned  value  into  their  entire  management  structure.  Also, 
Lockheed-Martin  Missiles  and  Space  has  announced  it  is  adopting  earned  value  for  all  its 
contracts.  These  include  contracts  with  military  or  commercial  customers. 
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Title:  Cost/Schedule  Control  System  Criteria:  A  lighthearted  introduction  for  the  non¬ 
believer 

Author:  Lee  R.  Lambert 
Format:  Article  in  AACE  Transactions 
Date/Published:  1988 
Pages:  D.ll.l-D.l  1.5 

The  author  generally  endorses  C/SCSC,  but  warns  that  in  order  for  it  to  be 
successful,  a  few  things  must  be  present.  First,  there  must  be  top  management  support 
for  the  implementation  of  a  C/SCSC  compliant  system.  Second,  education  and  training 
will  determine  if  the  system  is  productively  utilized  or  is  just  another  management  “toy”. 
The  rest  of  the  article  describes  how  to  set  up  an  effective  training  program. 

Several  benefits  of  the  C/SCSC  approach  are  given  including  the  ability  to: 

•  provide  information  that  will  enable  managers  and  contributors  to  take  a  more  active 
role  in  defining  and  justifying  their  own  roles. 

•  alert  managers  to  potential  problems  in  time  to  be  proactive  instead  of  reactive. 

•  allow  a  manager  to  clearly  demonstrate  timely  technical  accomplishments. 

•  provide  the  basis  for  significant  improvement  in  internal  and  external 
communications. 

•  provide  a  powerful  marketing  tool  for  future  technical  projects  and  programs  which 
require  high  management  content. 

•  provide  the  basis  for  consistent,  effective  management  system  based  training  and 
education. 
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•  provide  a  more  definitive  indication  of  the  cost  and  schedule  impact  of  technical 
problems. 

•  allow  tremendous  flexibility  in  its  application. 

Finally,  the  article  ends  by  stating  that  a  C/SCSC  validated  system  can  become  one  of  the 
most  meaningful,  effective  and  efficient  tools  a  technical  professional  has  available, 
given  that  the  users  are  informed  and  supportive. 
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Title:  The  Marginal  Cost  and  Practicality  of  Cost/Schedule  Control  Systems  Criteria 
Implementation 

Author:  Lampkin,  Eric  D. 

Format:  Comptroller  Research  Project 

Date/Published:  May  1992 

Pages:  42 

In  May  1991,  the  Joint  DoD/Industry  Total  Quality  Management  Team  (JTQMT) 
examined  and  identified  opportunities  to  improve  the  cost/schedule  management  process. 
The  JTQMT  gave  1 8  recommendations  for  streamlining  the  process,  surprisingly,  the  cost 
issue  was  not  the  most  dissatisfying  driver  against  C/SCSC  implementation.  Although  an 
almost  unanimous  agreement  that  the  cost  of  C/SCSC  implementation  and  cost 
performance  reporting  is  of  high  importance,  more  than  90%  of  all  interviewees  (DoD 
and  Industry)  placed  extremely  high  importance  on  the  need  for  C/SCSC. 

This  research  project  plans  to  expand  on  the  JTQMT  report  by  focusing  on  the 
cost  of  C/SCSC.  Specifically,  the  marginal  cost  and  practicality  of  C/SCSC 
implementation  will  be  examined  (marginal  C/SCSC  cost  is  the  cost  above  the  normal 
operating  costs  of  contractor  cost/schedule  management  systems).  The  project  eissumes 
that  the  cost  account  manager’s  fixed  salary  will  not  fluctuate  with  the  addition  or 
subtraction  of  administrative  tasks.  Secondly,  tremendous  savings  in  both  time  and 
money  can  be  realized  with  the  technieal  advancements  in  the  computer  hardware  and 
software  industry.  It  is  important  to  note  that  FAR  52.234-7001  states  “When  systems 
existing  at  the  time  of  contract  award  do  not  comply  with  the  criteria,  adjustments 
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necessary  to  assure  compliance  will  be  effected  at  no  change  in  contract  price  or  fee.” 
Although  the  FAR  implies  that  there  is  no  added  cost  to  the  Government,  the  contractor 
should,  and  does  receive  compensation  for  those  things  added  because  of  C/SCSC.  This 
compensation  is  negotiated  prior  to  contract  award. 

In  starting  the  project,  the  author  notes  that  it  is  possible  to  quantify  an  individual 
contractor’s  cost  for  C/SCSC  implementation,  system  maintenance,  and  use.  However, 
there  is  no  way  to  determine  a  quantified  dollar  value  that  can  be  applied  to  every 
government  contractor.  With  the  help  of  C/SCSC  consultants,  the  following  generic 
C/SCSC  implementation  process  was  designed  to  help  break-out  the  costs: 


Project  System  Implementation 

Contractor 

Person-Days 

Consultant 
Person-  Days 

Total  Person- 
Days 

Step  1  -  Team  organization  and  start  up 

40-60 

40-60 

Step  2  -  System  assessment 

128-188 

203-323 

Step  3  -  Subsystem  development 

275 

60-120 

180-300 

Step  4  -  System  documentation 

56-83 

143 

418 

Step  5  -  System  implementation 

130-200 

56 

112-139 

Step  6  -  System  training 

50-100 

180-300 

Step  7  -  Compliance  and  classification 
assistance 

20-40 

70-240 

Step  8  -  Additional  Support 

240 

38-58 

127-247 

TOTAL 

1008-1435 

382-432 

1390-1967 

To  test  the  validity  of  this  implementation  process,  an  implementation  process 
survey  was  designed  for:  1)  contractors  to  verify  those  activities  applicable  to  their 
implementation  process  and  those  associated  costs,  2)  capturing  those  activities  done 
solely  to  satisfy  the  C/SCSC,  3)  identifying  those  activities  that  would  be  eliminated  from 
day-to-day  management  if  the  C/SCSC  were  removed  from  the  contract,  and  4)  splitting 


104 


the  implementation  process  activities  and  their  associated  costs  into  recurring  and  non¬ 
recurring  activities.  Eighteen  surveys  were  returned;  eight  from  contractors,  and  ten  from 
experienced  AF  personnel. 

From  the  returned  surveys,  implementation  processes  were  validated,  and  costs 
were  divided  into  recurring,  and  non-recurring.  The  majority  of  contractors  revealed  that 
it  was  necessary  to  hire  consultants  at  contract  start-up  to  assist  them  in  the 
implementation  process.  All  of  the  respondents  indicated  that  to  satisfy  the  Government 
review  process,  they  established  subsystem  flow  diagrams  with  narratives,  and  that  they 
had  to  expand  their  written  procedures  to  describe  how  their  systems  complied  with  the 
criteria.  Additionally,  all  contractors  indicated  that  it  was  necessary  to  design  or  revise 
forms  to  help  comply  with  the  criteria.  Most  of  the  contractors  indicated  that  it  was 
necessary  to  purchase  new  hardware/software,  citing  the  large  volume  of  data  requiring 
compilation,  and  timely  reporting  as  the  primary  reasons.  Although  the  survey  identified 
those  activities  which  were  only  done  to  satisfy  the  criteria,  the  majority  of  contractors 
stated  that  they  would  continue  to  maintain  criteria  compliaint  systems  even  if  the  criteria 
were  removed  from  the  contract.  The  general  consensus  is  that  changes  in  the 
contractor’s  workload,  due  to  the  removal  of  the  criteria,  would  be  insignificant.  This 
conclusion  justifies  the  contractor’s  position  of  giving  nothing  back  to  the  Government 
for  the  removal  of  the  criteria. 

This  project  also  studied  the  actual  time  from  contract  award  to  system  validation. 
DoD  FAR  clause  52.234-7001  states  that  validation  is  to  occur  within  90  days  of  contract 
award.  While  this  would  be  ideal,  it  is  rarely  the  norm.  The  defense  industry  1 5-man  Ad 
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Hoc  Committee  recommends  that  validation  within  1 80  days  of  contract  award  would 
reflect  actual  practice.  The  Arthur  D.  Little  study  estimated  that  validation  process  takes 
12  months  for  a  subsequent  application  review  (SAR),  and  18  months  for  a  demonstration 
review.  From  the  surveys,  this  project  estimated  that  validation  takes  an  average  of  10.5 
months  to  complete.  This  project  did  note  that  assigning  the  implementation  process  to 
high-level  managers  versus  junior  personnel  was  the  most  significant  reason  for  shorter 
validation  times. 

Finally,  this  project  cited  other  studies  that  estimated  the  percentage  of  the 
contract  value  that  is  associated  with  implementing  C/SCSC.  A  private  study  by 
Humphreys  &  Associates  projected  a  cost  range  of  0.5%  to  4%  of  the  total  contract  value. 
This  estimate  was  further  refined  into  recurring  (SAR-type)  activities,  and  non-recurring 
(DEMO-type)  activities.  Their  study  estimated  the  recurring  costs  to  be  from  0.5%  to 
1 .5%  of  total  contract  value.  The  non-recurring  costs  were  estimated  as  1 .5%  to  2.5%  of 
total  contract  value.  The  MITRE  Corporation  study  for  Electronic  Systems  Division 
stated  that  the  C/SCSC  implementation  costs  for  software  acquisition  and  development 
are  0.1%  to  0.2%  of  the  total  contract  value.  A  cost  model  from  Decision  Plaiming 
Corporation  estimated  implementation  cost  as  0.6%  to  1%  of  the  total  contract  value.  A 
study  by  Major  Kouts  reported  estimates  from  0.5%  to  5%  of  total  contract  value,  with  a 
mean  of  3%.  The  DoD  Inspector  General  reported  that,  based  on  his  experience,  5%  of 
the  total  contract  value  was  a  reasonable  estimate  for  implementing  and  maintaining  a 
criteria  compliant  system.  The  following  table  summarizes  these  results: 
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Study 

Cost  Range* 

Basis 

Recurring 

Non-Recurring 

MITRE 

0.1  -0.2% 

Software  Acq  Model 

N/A 

N/A 

SPARROW 

0.6- 1.0% 

Cost  Est.  Model 

N/A 

N/A 

KOUTS 

0.5  -  5.0% 

Industry  Response 

N/A 

N/A 

DoDIG 

5.0% 

DoD  Response 

N/A 

N/A 

HUMPHREYS 

0.5%  -  4.0% 

Consultant  Response 

0.5%  - 1.5% 

2.5%  -  4.0% 

Avg  Range 

0.4%  -  2.86% 

0.4% 

1.63% 

Mean  of  Averages 

1.69% 

N/A 

N/A 

*  Percent  of  Negotiated  Total  Contract  Value 
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Title:  Survey  Relating  to  the  Implementation  of  Cost/Schedule  Control  Systems  Criteria 
Within  the  Department  of  Defense  and  Industry  -  Phase  I 

Author:  Arthur  D.  Little,  Program  Systems  Management  Company 
Format:  Report  for  the  Assistant  Secretary  of  Defense  (Comptroller) 

Date/Published:  5  December  1983 
Pages:  216 

This  report  was  conducted  in  an  effort  to  resolve  two  controversies  surrounding 
C/SCSC:  1)  do  the  costs  of  the  criteria  and  associated  reporting  requirements  outweigh 
their  benefits?,  and  2)  concerns  about  the  specifics  of  how  these  requirements  have  been 
implemented.  The  report  was  accomplished  in  two  phases,  the  first  being  a  mailed 
questionnaire,  and  the  second,  in-depth  interviews  with  DoD  and  industry  respondents. 
Four  populations  were  surveyed:  DoD  program  managers  and  business  managers,  and 
contractor  program  and  business  managers.  The  most  important  finding  of  the  survey 
was  that  the  majority  of  respondents  believe  that  C/SCSC  benefits  to  themselves 
outw  eigh  its  costs.  However,  the  majorities  ranged  from  53%  to  62%  in  the  four 
populations. 

Of  important  note  is  the  manner  in  which  the  report  investigated  the  question:  do 
the  benefits  of  C/SCSC  outweigh  its  costs?  This  report  notes  that  the  primary  difficulty 
in  answering  this  question  is  determining  the  incremental  cost  between  operation  of  a 
C/SCSC  compliant  system  and  the  operation  of  a  system  the  contractor  would  otherwise 


108 


use.  Since  information  of  the  cost  of  the  latter  is  not  readily  available,  this  report  dealt 
with  this  question  as  a  matter  of  respondent  perceptions. 

This  report  generated  the  following  data  from  the  specific  question, 
“Approximately  how  many  man  years  per  year  of  all  categories  of  employees  are  spent 
on  operating  your  C/SCSC-accepted  system  for  this  contract  and  analyzing  its  output?” 
(This  table  represents  the  approximations  provided  by  1 14  contract  business  managers); 


Cost  of  Operation  of 

C/SCSC-Accepted  System 

(Man-Years  per  Year) 

Contractor  Business 

Manager  Responses 

0-5 

18% 

6-10 

28% 

11  - 15 

21% 

16-20 

9% 

21-30 

11% 

31  or  More 

13% 

TOTAL 

100% 

Outside  of  this  qualitative  data,  the  issue  -  do  the  benefits  of  C/SCSC  outweigh 
the  costs,  was  determined  qualitatively. 

This  reports  find  four  primary  conclusions  about  the  benefits  vs.  Costs  of  C/SCSC 

issue: 
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1 .  C/SCSC-accepted  systems  are  considered  to  be  effective  in  assisting  to  control 
cost  and  schedule  performance. 

2.  CPRs  and  related  contractor  reports  are  considered  to  be  accurate.  CPRs  are 
considered  most  useful  for  determining  contract  cost  status. 

3.  C/SC  SC  benefits  are  considered  to  outweigh  the  costs.  Nevertheless,  most 
contractor  program  managers  see  a  need  for  minor  modifications  to  their  systems.  Many 
program  managers  agree. 

4.  Manpower  costs  for  operating  C/SCSC-compliant  systems  (not  the  cost  of 
C/SCSC)  is  approximated  as  1 0  man-years  per  year  or  less  for  nearly  half  of  contracts. 
Manpower  costs  appears  to  vary  directly  with  contract  value. 
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Title:  Survey  Relating  to  the  Implementation  of  Cost/Schedule  Control  Systems  Criteria 
Within  the  Department  of  Defense  and  Industry  -  Phase  II 

Author:  Arthur  D.  Little,  Program  Systems  Management  Company 

Format:  Report  for  the  Assistant  Secretary  of  Defense  (Comptroller) 

Date/Published:  15  August  1984 

Pages:  61 

Please  reference  the  abstract  on  Phase  I  prior  to  reading  this.  Phase  II  of  the 
Arthur  D.  Little  study  consisted  of  personal  interviews  in  an  attempt  to  gather  more 
detailed  information  on  the  issues  and  problems  suggested  in  the  Phase  I  results.  Four 
contractor  organizations  and  four  DOD  program  offices  within  each  of  the  three  military 
departments  were  selected  for  in-depth  interviews.  In  the  cost  and  benefits  of  C/SCSC 
portion  of  the  Phase  II  study,  the  objective  was  to  identify  the  minor  modifications  that 
program  managers  perceived  to  be  needed. 

These  interviews  lead  to  several  suggested  modifications  to  the  current  C/SCSC. 
The  most  prevalent  modification,  suggested  by  industry,  was  the  deletion  of  the  variance 
analysis  at  the  cost  account  level,  as  well  as  other  documented  analysis.  Another 
suggestion  was  the  need  to  raise  the  level  of  the  work  breakdown  structure  (WBS)  at 
which  cost  accounts  were  established.  A  final  industry  suggestion  was  the  deletion  of 
inappropriate  rules  related  to  replanning.  The  study  makes  an  important  note:  the 
government  need  for  information  is  not  always  taken  into  account  by  the  contractors. 

The  study  concluded  the  cost  and  benefit  portion  by  explaining  that  the  excessive  level  of 
detail  problems  are  due  to  the  inadequate  skills  and  insights  of  some  of  the  governmental 
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overseers.  These  overseers  include  both  government  personnel  on  the  demonstration  and 
subsequent  application  review  teams,  and  those  government  personnel  involved  with 
negotiating  WBSs,  reporting  levels,  and  thresholds  for  analysis. 

Overall,  Phase  II  concluded  that,  most  importantly,  C/SCSC  is  a  valid  concept 
and  approach  to  controlling  contract  performance.  Additionally,  the  study  presented  ten 
recommendations  for  improvement  which  are  listed  below: 

1 .  A  brief  White  Paper  discussing  the  purpose  and  value  of  C/SCSC 

2.  A  certification  process  for  practitioners 

3.  A  centrally  controlled  manpower  pool  should  be  established  for  technical 
support  and  augmentation  of  C/SCSC  review  and  surveillance  teams. 

4.  A  jointly  manned  C/SCSC  group  should  be  organized. 

5.  Use  of  incentives. 

6.  Initial  high-level  government  contacts  with  senior  contractor  official  for 
C/SCSC  reviews. 

7.  Schedule  event  discussion  on  CPR  problem  analysis  format. 

8.  Clarification  of  terminology  in  DOD  guidance. 

9.  Separate  submittal  of  CPR  formats  (i.e.  do  not  wait  for  all  formats  to  be 
completed,  send  out  the  formats  as  they  are  individually  finished). 

10.  Data-nets  for  CPRs  (automated  transmittal  of  CPR  data). 
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Title:  Cost/Schedule  Systems  Compendium 
Author:  National  Security  Industrial  Association 
Format:  Compendium 

Date/Published:  Washington,  DC:  October  1980 

This  compendium  summarizes  the  responses  from  74  companies  resulting  from  a 
questionnaire  developed  by  the  National  Security  Industrial  Association  (NSIA).  The 
purpose  of  this  project  was  to  provide  industry  with  a  data  base  to  aid  contractors  recently 
exposed  to  Performance  Measurement  System  Requirements  as  well  as  contractors  with 
previously  accepted  systems.  Another  objective  of  this  projects  was  to  provide  data  from 
which  future  improvements  could  be  made  in  requirements  and  implementation  practices. 

The  responses  to  several  of  the  questions  lend  evidence  to  the  merit  of  earned 
value.  For  instance,  when  asked  what  levels  of  senior  management  use  their  cost 
performance  reports  (with  their  earned  value  measurements)  for  status  information,  the 
respondents  indicated  the  report  is  utilized  by:  99%  of  contract  program  managers;  57% 
of  functional  branch  managers;  55%  of  finance  controllers;  50%  of  company  Vice 
Presidents;  as  well  as  very  small  portion  of  corporate  vice  presidents  and  other  managers. 

When  asked  what  measurements  are  visible  in  the  labor  work  package  system 
besides  budget,  the  respondents  indicated  that  “earned  value  credited”  is  used  more  than 
any  other  measurement  listed  including  “underlying  standards/goals”  of  the  organization. 

In  response  to  which  management  level  performs  the  analysis  and  makes  the 
routine  estimate  update  for  the  revised  EAC,  respondents  indicated  the  cost  account 
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manager  function  was  most  involved  with  these  tasks.  However,  there  was  strong 
indications  of  multiple  participation  by  various  levels  of  management. 

When  asked  to  evaluate  the  utility  of  their  system  procedures  to  satisfy  both 
internal  operating  needs  as  opposed  to  the  evaluation  needs  of  government  reviewers,  the 
results  indicated  that  73%  of  the  respondents  believe  that  C/SCSC  procedures  are  aligned 
to  satisf)'  operating  needs.  Further,  62%  indicated  C/SCSC  procedures  are  aligned  to 
satisfy  government  review  visibility. 

Another  part  of  the  survey  asked  which  C/SCSC  concepts  have  been  applied  to 
other  contracts  or  commercial  applications  where  C/SCSC  was  not  a  contractual 
requirement.  Although  responses  indicated  very  few  contractors  utilized  the  criteria  fully 
in  these  situations,  various  concepts  were  used  quite  heavily.  For  instance,  Budgeted 
Cost  of  Work  Performed  (BCWP)  was  utilized  by  over  70%  and  Estimate  At  Completion 
(EAC)  and  reporting  methods  by  over  80%  of  respondents. 
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Title:  Summary  Level  Variance  Analysis 


Author:  Pakiz,  John 
Format:  White  Paper 
Date/Published:  26  FEB  93 
Pages:  7 

This  white  paper  reflects  the  National  Security  Industrial  Association  (NSIA) 
initiative  to  officially  pursue  the  1992  C/SCSC  National  Workshop  recommendations. 
The  May  1991  Joint  Government  and  Industry  Report  for  Program  Management  on 
Cost/Schedule  Management  Process  detailed  the  most  significant  C/SCSC  improvements 
would  be  in  the  area  of  variance  analysis.  Both  Government  and  industry  cited 
excessively  detailed  requirements,  repetitive  narratives,  and  voluminous  paperwork  as 
reasons  for  dissatisfaction.  The  June  1992  DoD  Team  Chief  meeting  verified  that 
variance  analysis  reports,  written  at  the  cost  account  level,  a  feature  of  most  contractor 
systems,  are  not  required  by  policy.  These  cost  account  variance  reports  are  a  product  of 
unwritten  implementation  rules.  Significant  costs  can  be  avoided  ,  and  system 
effectiveness  improved  by  assuring  that  variance  analysis  reporting  is  a  function  of  how 
the  contractor  manages  the  work.  Summary  level  variance  analysis  includes  one  or  more 
cost  accounts,  while  still  identifying  the  problems,  impacts,  and  corrective  actions  for 
those  cost  accounts  with  significant  cost/schedule  variances.  Achievement  of  summary 
level  variance  analysis  must  be  done  on  a  case-by-case  basis  by  first  understanding  the 
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contractor’s  management  process,  and  then  determining  if  changing  the  system 
description  complies  with  C/SCSC. 

Three  contractor’s  assessments  of  the  savings  associated  with  summary  level 
variance  analysis  are: 

1 .  TRW  reports  a  cost  effective  approach  to  variance  analysis  resulting  in 
reduced  monthly  reporting  time,  reduced  steps  in  the  variance  analysis  cycle, 
and  elimination  of  redundant  reporting. 

2.  IBM  Federal  systems  cites  a  45%  reduction  in  variance  analysis  documents 
and  an  order  of  magnitude  reduction  in  the  volume  of  data  to  be  analyzed. 

3.  McDonnell  Douglas  Aerospace  discovered  that  fewer  reports  and  more 
efficient  and  shortened  analysis  cycles  were  the  result  of  the  switch  to 
summary  level  variance  analysis.  In  their  RDT&E  contract  test  case,  variance 
analysis  reports  were  reduced  from  100  to  15.  In  their  production  contract  test 
case,  variance  analysis  reports  were  reduced  from  70  to  6.  A  20%-40% 
reduction  in  cost  and  cycle  time  was  also  reported. 

In  conclusion,  this  white  paper  drafts  an  opportunity  to  reduce  the  cost  of 
C/SCSC,  while  the  compliance  of  the  criteria  is  still  fully  maintained. 


116 


Title:  What’s  the  value  of  Earned  Value? 


Author:  Nancy  Singletary 
Journal:  PM  Network 
Date:  December  1996 
Pages:  28-29 

This  author  talks  about  how  earned  value  has  evolved  over  the  years.  The 
predecessor  of  earned  value,  Cost/Schedule  Control  Systems  Criteria  (C/SCSC),  not  only 
has  survived  over  the  years  but  seems  stronger  and  more  useful  now  than  ever.  Earned 
value  is  no  longer  just  a  method  for  meeting  (C/SCSC)  standards  for  the  DOD.  The 
usefulness  of  the  earned  value  concept  is  reflected  by  its  increasing  use  in  commercial 
industries,  public  and  private  sectors,  manufacturing,  financial  institutions, 
pharmaceuticals,  and  other  industries.  The  increased  utility  of  earned  value  is  due  in  part 
to  advances  in  computers,  software  packages  and  documentation  to  support  the  earned 
value  methodology.  Earned  value  is  a  powerful  management  tool  that  is  essential  for 
risky  or  high-dollar  projects.  It  also  gives  us  the  ability  to  compare  the  progress  of 
dissimilar  projects. 
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Title:  Potential  Benefits  and  Problems  Related  to  Contractor  Compliance  with  DoDI 
7000.2  (C/SCSC) 

Author;  Whittenberg,  Ira  O. 

Format:  Article  in  National  Contract  Management  Journal 
Date/Published:  Fall  1972 
Pages:  73-80 

Despite  the  title  of  this  article,  the  author  focuses  more  heavily  on  the  problems 
related  to  contractor  compliance  with  C/SCSC.  However,  some  of  the  benefits  related  to 
compliance  with  C/SCSC  are  also  listed. 

First,  he  states  that  full  implementation  of  the  objectives  of  C/SCSC,  with 
reasonable  deviations  allowed  when  applicable,  will  result  in  improved  contractor 
management  systems,  better  cost  visibility  and  will  probably  help  reduce  overruns  on 
future  programs. 

Next,  through  complying  with  the  C/SCSC  requirements,  a  contractor  may  finally 
incorporate  long  needed  improvements  in  their  system  that  they  had  resisted 
implementing  before.  Further,  the  review  team  can  be  a  major  benefit  to  the  contractor 
by  suggesting  good  features  that  they  have  experienced  on  other  systems  in  the  defense 
industry. 

It  should  be  noted  that  Lieutenant  Colonel  Robert  R.  Kemps,  in  a  written  response 
to  the  editor  of  the  National  Contract  Management  Journal  rebukes  most  of  Whittenberg ’s 
criticisms  given  in  this  article. 


118 


Bibliography 


Abba,  Wayne.  “Earned  Value  Management  -  Reconciling  Government  and  Commercial 
Practices,”  Program  Manager,  26:  58-69  (January-February  1997). 

Abba,  Wayne.  “The  A- 12  Program  Management  Summary,”  From  presentation  at 
NSIA/MSS  Meeting.  January  1991. 

Abba,  Wayne.  “Cost/Schedule  Control  Systems  Criteria  White  Paper,”  Program 
Manager,  15:  45-47  (November-December  1986). 

Beach,  Chester  Paul,  Jr.  A-I2  Administrative  Inquiry.  Report  to  the  Secretary  of  the 
Navy.  Washington:  Department  of  the  Navy.  1990. 

Biery,  Frederick  P.  “Cost  Growth  and  the  Use  of  Competitive  Acquisition  Strategies.” 
Paper  Presented  at  the  National  Estimating  Society  Conference.  1985. 

Brodkorb,  Richard,  Decision  Planning  Corporation.  The  Cost  of  the  "Criteria.  ”  A 

Report  Presented  to  Air  Force  Institute  of  Technology  Course  362.  11  February 
1992. 

Christensen,  David  S.  Professor  of  Accoimting,  Air  Force  Institute  of  Technology, 
WPAFB  OH.  Personal  Interview.  1  July  1997. 

Christensen,  David  S.  “Project  Advocacy  and  the  Estimate  at  Completion”,  Journal  of 
Cost  Analysis,  ISSN:  0882-3871:  35-60  (Spring  1996). 

Christensen,  David  S.  “Determining  an  Accurate  Estimate  at  Completion.”  National 
Contract  Management  Association  Journal,  25:  17-25  (Spring  1993). 

Christensen,  David  S.  “Control  and  analysis  of  performance  measurement  data.” 
Monograph  -  Air  Force  Institute  of  Technology.  January  1992. 

Christensen,  David  S.  “Management  Control  Systems  Theory  and  C/SCSC,”  National 
Estimator,  8:  29-34  (Fall  1989). 

Christensen,  David  S.  and  Scott  Heise.  “Cost  Performance  Index  Stability”  National 
Contract  Management  Association  Journal,  25:  7-15  (Spring  1993). 


119 


Coopers  &  Lybrand  and  TASC.  The  DoD  Regulatory  Cost  Premium:  A  Quantitative 

Assessment.  A  Report  for  Dr.  William  J.  Perry,  Secretary  of  Defense.  December 
1994. 

Defense  Acquisition  Deskbook.  Version  1.4.48.  Computer  Software.  Defense 
Acquisition  Deskbook  Joint  Program  Office,  WPAFB  OH,  1997. 

Defense  Contract  Management  Command  (DCMC).  Earned  Value  Implementation 
Guide.  11  December  1996. 

Department  of  Defense  (DOD).  DOD  5000.2-R.  15  May  1996. 

Department  of  Defense  (DOD)  and  National  Security  Industrial  Association  (NSIA). 

Cost/Schedule  Management  Process.  Report  for  Program  Management.  17  May 
1991. 

“Feedback  -  The  Cost  of  C/SCSC,”  Program  Manager ’s  Newsletter:  22  (January- 
February  1978). 

Financial  Accounting  Standards  Board  (FASB).  Statement  of  Financial  Accounting 
Concepts.  Homewood  IL:  Irwin,  1993. 

Fleming,  Quentin  W.  Cost/Schedule  Control  Systems  Criteria  -  A  Management  Guide  to 
C/SCSC.  (Revised  Edition).  Chicago:  Probus  Press  Inc,  1992. 

Fleming,  Quentin  W.  and  Joel  M.  Koppelman.  Earned  Value  Project  Management. 
Upper  Derby  PA:  Project  Management  Institute,  1996. 

Fleming,  Quentin  W.  and  Joel  M.  Koppelman,  C&SM/C.2-Linking  Contractor  Payments 
to  Contractor  Performance,  39*  Annual  Meeting,  25-28  June  1995,  AACE 
International  209  Prairie  Avenue,  Suite  100  Morgantown  WV  26505  ISBN  1- 
885517-01-7. 

Gadeken,  Owen  C.  and  Thomas  S.  Tison.  “The  Cost  of  C/SCSC,”  Program  Manager, 
12:  13-18  (July- August  1983). 

General  Accounting  Office  (GAO),  National  Security  and  International  Affairs  Division. 
Significant  Changes  in  DOD’s  Earned  Value  Management  Process.  5  May  1997. 

General  Accounting  Office  (GAO),  National  Security  and  International  Affairs  Division. 
“Efforts  to  Reduce  the  Cost  to  Manage  and  Oversee  DOD  Contracts.”  1 8  April 
1996. 


120 


Grskovich,  D.  L.  “What  is  C/SCSC?-In  English,  Please!”  National  Contract 
Management  Journal,  23:  25-32  (Winter  1990). 

Hatfield,  Michael  A.  “The  Case  for  Earned  Value,”  PMNETwork:  25-27  (December 
1996). 

Industrial  Performance  Management  Council  (IPMC)  Homepage,  WWWeb, 
http;//www.acq.osd.mil/pm/intemat/ipmc/ipmc.html  (15  Jul  97). 

Industry  Standard  Guidelines  for  Earned  Value  Management  Systems.  NSIA 

Management  Systems  Subcommittee  EVMS  Work  Team.  8  August  1996. 
WWWeb,  http://www.acq.osd.mil/pm/nevvpolicy/indus/evms_gde.htm  (14  Jan 
97). 

Lampkin,  Eric  D.  “The  Marginal  Cost  and  Practicality  of  Cost/Schedule  Control  Systems 
Criteria  Implementation.”  Comptroller  Research  Project,  SAF/FM,  Washington 
DC.  May  1992. 

Little,  Arthur  D.  Inc.  Survey  Relating  to  the  Implementation  of  Cost/Schedule  Control 
Systems  Criteria  Within  the  Department  of  Defense  and  Industry  -  Phase  II.  A 
Report  for  The  Assistant  Secretary  of  Defense  (Comptroller).  15  August  1984. 

Little,  Arthur  D.  Inc.  Survey  Relating  to  the  Implementation  of  Cost/Schedule  Control 
Systems  Criteria  Within  the  Department  of  Defense  and  Industry  -  Phase  I.  A 
Report  for  The  Assistant  Secretary  of  Defense  (Comptroller).  5  December  1983. 

National  Security  Industrial  Association  (NSIA).  Cost/Schedule  Systems  Compendium. 
Prepared  by  the  NSIA  Management  Systems  Subcommittee,  Washington  DC. 
September  1980. 

Office  of  Management  and  Budget  (0MB),  Principles  of  Budgeting  for  Capital  Asset 
Acquisitions.  Washington:  Government  Printing  Office,  1987. 

Pakiz,  John.  “Summary  Level  Variance  Analysis.”  White  Paper,  National  Security 
Industrial  Association  (NSIA),  Washington  DC.  26  February  1993. 

Presutti,  A.  H.  Cost/Schedule  Control  Systems  Criteria:  A  Government  Process  That 
May  Work.  Government  Accountants  Journal:  47-54  (Spring  1993). 

Singletary,  Nancy.  “Whaf  s  the  Value  of  Earned  Value?”  PM  Network,  December  1996: 
28-30. 


121 


Whittenberg,  Ira  O.  Potential  Benefits  and  Problems  Related  to  Contractor  Compliance 
with  Department  of  Defense  Instruction  7000.2  Cost/Schedule  Control  Systems 
Criteria  (C/SCSC),  National  Contract  Management  Journal,  6:  No.  2,  73-80  (Fall 
1972). 


Vita 


Captain  John  R.  Cole  Jr.  was  bom  on  13  November  1971  in  Hartford,  Wisconsin. 
He  graduated  from  Hartford  Union  High  School  in  1989  and  entered  imdergraduate 
studies  at  the  United  States  Air  Force  Academy  in  Colorado  Springs,  Colorado.  He 
graduated  with  a  Bachelor  of  Science  degree  in  Management  and  received  his 
commission  in  June  1993. 

His  first  assignment  was  at  Eglin  AFB  as  a  financial  management  officer  at  the 
AMRAAM  JSPO.  In  May  1996,  he  entered  the  School  of  Logistics  and  Acquisition 
Management,  Air  Force  Institute  of  Technology.  Upon  graduation,  he  will  be  assigned  to 
the  Space  and  Missile  Center,  Los  Angeles  AFB,  California. 


Permanent  Address:  622  East  Monroe  Ave  Apt  #2 
Hartford,  WI  53027 


123 


Vita 


Lt  Judson  (Judd)  M.  Fussell  was  bom  on  19  Jun  1965  in  Boaz,  Alabama.  He 
graduated  from  Crossville  High  School  in  1983  and  began  his  enlistment  with  the  Air 
Force  in  February  1984.  At  the  completion  of  his  four  year  enlistment,  he  separated  from 
the  Air  Force  to  attend  college  full  time.  He  graduated  with  a  Bachelor  of  Science  degree 
in  Marketing  Management  from  the  University  of  Alabama  in  May  1992.  He  received 
his  commission  on  22  June  1994  upon  graduation  from  Officer  Training  School. 

His  first  assignment  as  a  financial  manager  was  at  the  F-16  System  Program 
Office  (SPO)  on  Wright-Patterson  AFB,  Ohio.  In  May  1996,  he  entered  the  School  of 
Logistics  and  Acquisition  Management,  Air  Force  Institute  of  Technology.  Upon 
graduation,  he  will  be  reassigned  to  the  Space  and  Missile  Center,  Los  Angeles  AFB, 
California. 


Permanent  Address:  50  Momingview  Drive 
Boaz  AL  35957 


124 


REPORT  DOCUMENTATION  PAGE 


Form  Approved 
0MB  No.  074-0188 


Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  the  collection  of  information,  including 
suggestions  for  reducing  this  burden  to  Washington  Headquarters  Services.  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202-4302, 
and  to  the  Office  of  Management  and  Budget,  Paperwork  Reduction  Project  (0704-0166),  Washington.  DC  20503 _ _ 


1.  AGENCY  USE  ONLY  (Leave  2.  REPORT  DATE  3.  REPORT  TYPE  AND  DATES  COVERED 

blank)  September  1997  Master’s  Thesis 


4.  TITLE  AND  SUBTITLE 

A  COST-BENEFIT  ANALYSIS  OF  EARNED  VALUE  MANAGEMENT 
SYSTEM  CRITERIA 

6.  AUTHOR(S) 

CAPTAIN  John  R.  Cole  Jr. 

1*^  LIEUTENANT  Judson  M.  Fussell 


7.  PERFORMING  ORGANIZATION  NAMES(S)  AND  ADDRESS(S) 

Air  Force  Institute  of  Technology 

2750  P  Street 

WPAFB  OH  45433-7765 


8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 

AFIT/GCA/LAS/97S-2 


9.  SPONSORING  /  MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Mr.  Wayne  Abba,  OUSD  (A&T)  API/PM  Room  3E1025 
Director  Acquisition  Program  Integration 
3020  Defense  Pentagon 
Washington  DC  20301-3020 


11.  SUPPLEMENTARY  NOTES 


10.  SPONSORING  /  MONITORING 
AGENCY  REPORT  NUMBER 


12a.  DISTRIBUTION  /  AVAILABILITY  STATEMENT 


Approved  for  release;  distribution  unlimited 


12b.  DISTRIBUTION  CODE 


13.  ABSTRACT  (Maximum  200  Words)  In  December  1996,  the  Cost/Schedule  Control  Systems  Criteria  (C/SCSC)  was 
officially  replaced  by  the  Earned  Value  Management  Systems  (EVMS)  criteria.  The  switch  to  EVMS, 
coupled  with  current  acquisition  reform  changes,  have  left  many  wondering  what  the  effects  of  these 
changes  will  be.  This  thesis  defines  the  costs  and  benefits  of  the  old  C/SCSC,  and  then  compares  them. 
Additionally,  this  thesis  discusses  the  changes  accompanying  the  switch  to  EVMS  and  the  effect  on  the 
costs  and  benefits.  The  marginal  costs  of  C/SCSC  are  defined  as  the  difference  between  the  costs  of  a 
C/SCSC-compliant  system  and  a  contractors  ‘normal’  management  control  system.  The  marginal  system 
compliance  costs  are  334  -  481  person  days,  while  the  marginal  operating  costs  are  50%  of  the  C/SCSC- 
compliant  operating  costs.  Fourteen  benefits  of  C/SCSC  are  detailed  in  this  thesis.  The  most  important 
benefit  discovered  was  the  data  reliability  that  comes  with  a  criteria-compliant  management  control  system. 
The  main  difference  between  C/SCSC  and  EVMS  is  the  system  certification  process.  Under  C/SCSC,  DoD 
teams  would  have  to  certify  a  contractor’s  system.  Under  EVMS,  contractors  have  the  ability  to  self-certify 
their  system  (with  final  government  approval).  Cost  savings  may  result  through  self-certification  without 
reductions  in  the  benefits. 


14.  Subject  Terms 

C/SCSC,  MANAGEMENT  PLANNING  and  CONTROL,  VALIDATION,  CRITERIA 


17.  SECURITY  CLASSIFICATION 
OF  REPORT 

UNCLASSIFIED 


NSN  7540-01-280-5500 


18.  SECURITY  CLASSIFICATION 
OF  THIS  PAGE 

UNCLASSIFIED 


19.  SECURITY  CLASSIFICATION 
OF  ABSTRACT 

UNCLASSIFIED 


15.  NUMBER  OF  PAGES 

134 


16.  PRICE  CODE 


20.  LIMITATION  OF  ABSTRACT 


Standard  Form  298  (Rev.  2-89) 

Prescribed  by  ANSI  Std.  Z39-18 


AFIT  Control  Number  AFIT/GCA/LAS/97S-2 


AFIT  RESEARCH  ASSESSMENT 

The  purpose  of  this  questionnaire  is  to  determine  the  potential  for  current  and  future  applications 
of  AFIT  thesis  research.  Please  return  completed  questionnaire  to:  AIR  FORCE  INSTITUTE 
OF  TECHNOLOGY/LAC,  2950  P  STREET,  WRIGHT-PATTERSON  AFB  OH  45433-7765. 
Your  response  is  important.  Thank  you. 

I.  Did  this  research  contribute  to  a  current  research  project?  a.  Yes  b.  No 


2.  Do  you  believe  this  research  topic  is  significant  enough  that  it  would  have  been  researched  (or 
contracted)  by  your  organization  or  another  agency  if  AFIT  had  not  researched  it? 

a.  Yes  b.  No 


3.  Please  estimate  what  this  research  would  have  cost  in  terms  of  manpower  and  dollars  if  it  had 
been  accomplished  under  contract  or  if  it  had  been  done  in-house. 

Man  Years _  $ _ 

4.  Whether  or  not  you  were  able  to  establish  an  equivalent  value  for  this  research  (in  Question 
3),  what  is  your  estimate  of  its  significance? 

a.  Highly  b.  Significant  c.  Slightly  d.  Of  No 

Significant  Significant  Significance 

5.  Comments  (Please  feel  free  to  use  a  separate  sheet  for  more  detailed  answers  and  include  it 
with  this  form): 


Name  and  Grade 


Organization 


Position  or  Title 


Address 


